Multimodal Generative Models for Scalable Weakly-Supervised Learning
Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weaklysupervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations-edge detection, colorization, segmentation-as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

This paper presents a generative approach to multimodal deep learning based on a product-of-experts (PoE) inference network. The main idea is to assume the joint distribution over all modalities factorises into a product of single-modality data-generating distributions when conditioned on the latent space, and use this to derive the structure and factorisation of the variational posterior. The proposed model shares parameters to efficiently handle any combination of missing modalities, and experiments indicate the model’s efficacy on various benchmark datasets.  The idea is intuitive, the exposition is well-written and easy to follow, and the results are thorough and compelling. I have a few questions / comments, mainly about the relationship of this work with respect to previous approaches ([15] and [21] in the text).  Firstly, the following line in the introduction is incorrect: “While fully-supervised deep learning approaches [15, 21] can learn to bridge modalities, generative approaches promise to capture the joint distribution across modalities and flexibly support missing data”. Both [15] and [21] are unsupervised generative approaches used to model the joint distribution, and have the machinery to deal with missing modalities. Furthermore, [21] employs a restricted Boltzmann machine, which is itself a product-of-experts model, so I would like to see a better justification for how the PoE is incorporated in the proposed approach, and how this PoE differs from [21].  Secondly, the sub-sampled training paradigm in section 2.2 is also quite similar to the denoising training approaches used in [15] and [21] (which drop out modalities randomly), so I think this could be acknowledged and the proposed approach contrasted. Lines 106-107 are also a bit unclear: is the procedure to compute the ELBO using either (1) all modalities, (2) a single modality, or (3) k randomly selected modalities; each with equal probability? I would rewrite that sentence for clarity.  Lastly, two related points: (1) while the approach is benchmarked against BiVCCA, JMVAE, and CVAE, there are no comparisons some of the less recent approaches to multimodal deep learning ([15] and [21]); (2) while the experiments are comprehensive on MNIST and CelebA, these benchmarks are a little contrived in the multimodal learning case, and I believe the paper would be strengthened with a result on a more real-world multimodal problem. An additional experiment on the MIR Flickr dataset would address both of these points.  Minor nitpicks: - Figure 1 caption talks about PoE, but the diagram refers to PoG. - Line 126, I would change “we’ve argued...” to a more formal tone. Ie. “we have previously argued in Section 2...”  [15] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696, 2011. [21] Nitish Srivastava and Ruslan R Salakhutdinov. Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222–2230, 2012.   --AFTER REBUTTAL-- I have read the authors' response and other reviews and my overall assessment has not changed, but there is one critical aspect that should be addressed.  After discussions with the AC and reviewers, we believe the claim in line 73 is not correct - the correct posterior p being factorised doesn't necessarily mean the optimal approximate posterior q when using a given family of distributions for q will also be factorised. I the argument should be changed to indicate the factorisation of p was inspiration for the form of q rather than claiming the "correct" q must be factorised, and making the argument specific to the multi-modality application as well.  I have a number of other comments that I feel should be addressed before the final version: 1) The authors recognise that comparison to an RBM is interesting (given it is also a PoE) and promise a new RBM baseline in the final draft. To clarify, I think this should be the same deep architecture, trained with a shared RBM representation, rather than just a single layer RBM, to make it a fair comparison (something like [21]. Obviously the proposed approach is end-to-end and does not require layer-wise training like a Deep Boltzmann machine, an important distinction that should be emphasised.  2) The response document states that "the shared representation in [15] is not a probabilistic measure", but this does not prevent it from being a generative model as the modality-specific latent representations are stochastic and it is still modelling the data-generating distribution. As such, I think it would be incorrect to state (as in the original draft), that [15] and [21] are not generative models. I believe this should be rectified for both, not just for [21]. 3) The additional text translation task is compelling - but I assume that the baseline performances will be reported on this task as well? 4) My comment on the sub-sampling training of the model was, I believe, only partially addressed. In particular, I still think the section detailing this process could be written more clearly, in terms of what data is used and how this relates to previous "denoising training" procedures (eg. [15] and [21]), which I believe are effectively sub-sampling in any case.  All in all, I think this would be a good addition to the conference.


Scaling provable adversarial defenses
Recent work has developed methods for learning deep network classifiers that are provably robust to norm-bounded adversarial perturbation; however, these methods are currently only possible for relatively small feedforward networks. In this paper, in an effort to scale these approaches to substantially larger models, we extend previous work in three main directions. First, we present a technique for extending these training procedures to much more general networks, with skip connections (such as ResNets) and general nonlinearities; the approach is fully modular, and can be implemented automatically (analogous to automatic differentiation). Second, in the specific case of ∞ adversarial perturbations and networks with ReLU nonlinearities, we adopt a nonlinear random projection for training, which scales linearly in the number of hidden units (previous approaches scaled quadratically). Third, we show how to further improve robust error through cascade models. On both MNIST and CIFAR data sets, we train classifiers that improve substantially on the state of the art in provable robust adversarial error bounds: from 5.8% to 3.1% on MNIST (with ∞ perturbations of = 0.1), and from 80% to 36.4% on CIFAR (with ∞ perturbations of = 2/255).

__ Post-rebuttal feedback: I thank the authors for the clarification. Based on the rebuttal letter, in the final version I'd suggest emphasizing the provable defense is guaranteed in probabilistic sense. Even though I agree in test time the geometric estimator is not necessary, what you indeed certified are training data, instead of test data. This is a nice piece of work and I enjoy reading it. __ This paper primarily builds upon the seminal work in [Wong-Kolter’17] to show improved and generalized verification and model robustness evaluation in three aspects: (I) extension to general networks (e.g., those with skip connections) and non-ReLU activation functions; (II) make the proposed verification method scalable via a nonlinear random projection technique, exhibiting linear computation complexity in the input dimension and the number of hidden units; (III) Verifiable bound improvement using model cascades. In my opinion, this work has made important contributions in norm-bounded robustness verification by proposing a scalable and more generic toolkit for robustness certification. The autodual framework is both theoretically grounded and algorithmically efficient. However, I also have two major concerns about this work: (I) the proposed nonlinear random projection leads to an estimated (i.e., probabilistic) lower bound of the minimum distortion towards misclassification, which is a soft robustness certification and does not follow the mainstream definition of deterministic lower bound; (II) Since this method yields an estimated lower bound, it then lacks performance comparison to existing bound estimation methods.   Strength: 1. This paper is well-written and it proposes a theoretically grounded and practically important solution to support scalable robustness verification of neural networks with different network architectures (e.g., skip connections) and non-ReLU activation functions.  2. To the best of my knowledge, this work is the first verification method that can scale up to more than 10K hidden units within a reasonable computation time.  Weakness: 1. An estimated lower bound (on minimum distortion for misclassification) is not a lower bound: I was very concerned that the current write-up may give a false sense of scalable “deterministic” defense. In order to speed up the computation, the use of the random projection makes the lower bound an estimation rather than a hard certificate. However, the notion of statistical lower bound (that means with some probability the result is not a lower bound) actually differs from the mainstream definition of verification – which needs to be deterministically verified, such as the definition in [Wong-Kolter’17]. Although the authors have discussed in the appendix on how to convert this probabilistic bound to the associated lower bound for certification, I am not quite convinced that one should call the proposed method a “provable defense”. I suggest modifying the title to be "Scaling adversarial defenses with statistical guarantees". The use of "provable" here is very vague and may be messed up with deterministic robustness certification. In the introduction, the authors should also highlight that the notion of robustness certification in this paper is statistical rather than deterministic. 2. Lack of performance comparison to existing methods: Since the proposed scalable approach yields a statistical estimate of a lower bound, there are already existing works that use extreme value theory to estimate the a lower bound, which is scalable to deep ImageNet classifiers and empirically aligned with the results of strong attacks (i.e., tight upper bounds). Due to the same spirit of efficient statistical estimation of a lower bound, I suggest the authors to compare with [R1] by fixing the same network and setting the \epsilon in Table 2 to be the same as the estimated lower bound in [R1] (by tuning ), and compare the resulting robust errors. 3. Extension beyond ReLU: One of the main contributions claimed in this work is the capability of certifying robustness for networks with non-ReLU activation functions. However, the experimental results are all based on ReLU networks and in the manuscript the authors merely refer readers to [Dvijotham et al., 2018]. I suggest including additional descriptions on the detailed procedures of incorporating the results in [Dvijotham et al., 2018] into the autodual framework and provide empirical results on non-ReLU networks for proper justification.  Summary:  This paper has made important contributions in scaling up the robustness certification method. However, the proposed method trades in the commonly used deterministic robustness guarantee to a probabilistic one (the latter is not a mainstream definition of certified robustness under norm-bounded perturbation attacks), and lacks performance comparison to existing methods that also use statistical estimates of lower bounds. Therefore, my main concerns are the credibility of probabilistic robustness guarantee and lacking performance comparison to other statistical robustness certification approaches.  [R1] Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach


Mallows Models for Top-k Lists
The classic Mallows model is a widely-used tool to realize distributions on permutations. Motivated by common practical situations, in this paper, we generalize Mallows to model distributions on top-k lists by using a suitable distance measure between top-k lists. Unlike many earlier works, our model is both analytically tractable and computationally efficient. We demonstrate this by studying two basic problems in this model, namely, sampling and reconstruction, from both algorithmic and experimental points of view.

The authors consider the sample generation problem, center (location permutation) construction, and learning mixture model parameters for Mallows models with a top-k distance function. While most proposals for top-k lists are not invariant to tail (i.e., n-k) ordering, require generating the entire permutation, and/or analyze this from the repeated insertion model (RIM) perspective, this work defines a top-k distance function (ostensibly the distance function presented in [Klementiev, Roth & Small; ICML08] from [Fagin, et al].) and analyze the resulting top-k Mallows model in depth. For generation, exact sampling is O(k^2 4^k _ k^2 log n) and an approximate Markov chain sampler is O(k^5 log k) — which is important as even though computationally intensive, does not include n as a term, which is crucial when k is generally in the 10s and n can be in the millions. Center permutation construction and learning mixtures are also computationally efficient with provable guarantees. The Markov chain based approach for generation is used to generate synthetic data, which is then used for center permutation (re)construction, which is contrasted with Borda count and a second simple baseline — showing greater likelihood of returning the center. Finally, they consider a dataset of top-10 movie lists from the Criterion collection and return the center permutation.   In summary, the problem studied in this paper is important and several theoretical findings are explained — thus, I believe it is likely that it will have some impact in the ‘rank aggregation’ community. However, the presentation and general writing is lacking, even if I think this is partially an effect of it really being a longer theoretical contribution that is compressed into conference format. Secondly, the experiments are underwhelming — the synthetic experiments perform exactly as expected and the Criterion dataset experiment doesn’t provide any meaningful evidence that this is useful. Given that there are multiple papers in this vein that have compelling experiments, a stronger paper would have empirical comparison (even possibly in terms of runtime in addition to performance) that contrasts this with the proposed method.  Below is the paper evaluated with respect to the requested dimensions:  [Quality]: The theoretical contribution is through and relatively compelling. By defining a different distance function, the authors provide a clever method for analysis that emits useful results and may be utilized by others for other potential structure aggregations. For example, with a suitable distance function for sequences, trees, and other structures, this is a more useful analytical framework than a RIM-style analysis. Conversely, the empirical evaluation is relatively weak. While this isn’t really the point of the paper, a stronger empirical analysis is possible and would make for a much more compelling paper (i.e., some of the examples given in the introduction). As the Criterion collection is mostly masterpieces, the resulting center permutation says almost nothing.  [Clarity]: The paper is relatively clear for those who have worked in this area, but is very dense for more casual readers. For example, the ‘p-parameterized’ discussion in lines 66-70 doesn’t make intuitive sense despite them providing an example. It isn’t until one draws a picture based on line 128-134 that the example even makes sense. Secondly, while space is at a premium, the paper is largely unreadable past certain points without referring to the appendices. It would be nice if some sketches or intuitive explanation for the general analysis method could be presented. Finally, there is a general sloppiness to the paper (that could be fixed for a camera-ready).  Line 4: work -> works Line 23: mode -> model Line 32: top -> top-k (twice) Line 51: [same] Line 182: section -> section, (comma) Line 284: top—k -> top-k (There are more, but this is just what I circled in the first reading)  The paper can be tightened in some sections (i.e., motivation) to given intuition and improve experiments and experiment discussion. It also doesn’t help that the experiment figures are essentially unreadable without zooming in on the PDF file.  [Originality]: People have clearly been thinking about top-k lists and (unless I am misunderstanding), this is the distance function from [KRS08], even if they do not analyze in nearly the same depth — as this wasn’t the point of that paper. Thus, the strength of this paper is in the execution of solving a ‘sort of known’ difficult open problem as opposed to being particularly original.  [Significance]: As stated above, the theoretical results are reasonably compelling and the proof methods have potential to inspire other work. Thus, while potentially a bit esoteric, it has the potential to be an influential work. I enjoyed reading the paper and it gave me some ideas on how to analyze some of the problems I have been thinking about related to these problems. Conversely, without stronger empirical evidence, they are likely to reach a smaller audience.   Overall, nice results -- but the paper needs work in general. Thus, I am inclined to 'weak reject' in its current form.  Comments after author feedback: After reading the other reviews and particularly the rebuttal (and paper again), I am much more inclined to accept. This is an important problem with interesting results. I would still encourage the authors to tighten up the writing to make it more approachable (even if it is essentially a math paper).


Image Inpainting via Generative Multi-column Convolutional Neural Networks
In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.

Summary This submission tackles the problem of image inpainting. It adapts the idea of multi-scale predictions by running three branches that predict features at different scales which are concatenated before two convolutions create the final prediction. The loss consists of three components. The relatively common L1 and local+global discriminator loss and lastly a novel implicitly diversified MRF that correlates patches of features between the ground truth and the prediction. The method is evaluated on five diverse datasets and with several ablation studies analyzing the influence of different parts.  Strengths  - The ablation study shows the contribution of the different components well. I am quite surprised about the added detail through the increased receptive field.   - The method shows impressive image quality on a wide range of datasets.  - The paper is technically sound, well structured and easy to follow.  - The fact that the expensive MRF optimization can be pushed to the training and is not needed for evaluation is elegant and makes the model much more usable during inference.   Weaknesses  - Since it is difficult to evaluate image generation tasks quantitatively, a human study could be performed where users need to select their preferred generated image. This evaluation could help showing the increased visual quality with respect to other methods.  - Since Equation 5 is applied multiple times it should contain some notion of iterations (e.q. M_w^{(i)}. Otherwise M_w appears on both sides of the equation.   - An interesting evaluation could be correlating reconstruction quality and RS measure. This could show that the loss actually helps finding the right correspondences between patches.  Minor Comments  - L. 82: {1, 2, …, n}  - The type of upsampling for the feature maps before concatenation is not mentioned.   - Since the ID-MRF is only used in training it would be interesting to report training times or even time per image during training compared to inference.   - Potential typo in Equation (4). L_M(conv4_2) appears twice.   - The intuitive example for L_M from the supplementary material could be included in the paper. I found it easier to follow than the explanations in lines 117-128.  - L. 224: our method is still difficult to deal with large-scale -> still has difficulties dealing with […]   Rating & Evaluation Given the above strengths and weaknesses I am in favor of accepting this submission. A user study and a more in-depth analysis of the proposed MRF loss could make this paper even stronger.   --  After reading the other reviews and the rebuttal I find most of my concerns addressed. The authors provide several additional quantitative results including a user study with convincing outcome. I vote in favor of accepting the paper.


Hierarchical Graph Representation Learning with Differentiable Pooling
Recently, graph neural networks (GNNs) have revolutionized the field of graph representation learning through effectively learned node embeddings, and achieved state-of-the-art results in tasks such as node classification and link prediction. However, current GNN methods are inherently flat and do not learn hierarchical representations of graphs-a limitation that is especially problematic for the task of graph classification, where the goal is to predict the label associated with an entire graph. Here we propose DIFFPOOL, a differentiable graph pooling module that can generate hierarchical representations of graphs and can be combined with various graph neural network architectures in an end-to-end fashion. DIFFPOOL learns a differentiable soft cluster assignment for nodes at each layer of a deep GNN, mapping nodes to a set of clusters, which then form the coarsened input for the next GNN layer. Our experimental results show that combining existing GNN methods with DIFFPOOL yields an average improvement of 5-10% accuracy on graph classification benchmarks, compared to all existing pooling approaches, achieving a new state-of-the-art on four out of five benchmark data sets.

the paper introduces a differentiable clustering operation enabling neural methods acting on graphs to process information in a hierarchical fashion in analogy to CNN pooling layers (but preserving graph symmetry properties).  The paper is very well written, the method is simple (which is not a bad thing!) and widely applicable, and the results are favourable. Building hierarchical representations is an important feature of CNNs and extending this to more general graph datatypes is clearly an important contribution. Overall I strongly support publication of this paper.  I only have minor additional thoughts:  1. When the edges have labels (e.g. single/double/triple bonds in the chemistry example used in the motivation of the method), would equation (4) be extended to transform adjacency matrices for each edge type separately by S? In this case the link prediction objective will need altering. Maybe including an experiment with labelled edges would make the presentation of this otherwise simple method more complete?  2. You have to manually set the maximum number of clusters in each layer (please make this clearer near equation 6 - currently the only explicit reference to this fact is at the end of the caption of Fig2). It is nice that the GNN can learn to use less than the maximum number of clusters, but how sensitive are the results to the cluster count hyperparameters? Are the examples in Fig2 compelling because there is obvious clustering in the graphs? What does the clustering figure look like for data where the clusters are not so obvious?


DifNet: Semantic Segmentation by Diffusion Networks
Deep Neural Networks (DNNs) have recently shown state of the art performance on semantic segmentation tasks, however, they still suffer from problems of poor boundary localization and spatial fragmented predictions. The difficulties lie in the requirement of making dense predictions from a long path model all at once, since details are hard to keep when data goes through deeper layers. Instead, in this work, we decompose this difficult task into two relative simple sub-tasks: seed detection which is required to predict initial predictions without the need of wholeness and preciseness, and similarity estimation which measures the possibility of any two nodes belong to the same class without the need of knowing which class they are. We use one branch network for one sub-task each, and apply a cascade of random walks base on hierarchical semantics to approximate a complex diffusion process which propagates seed information to the whole image according to the estimated similarities. The proposed DifNet consistently produces improvements over the baseline models with the same depth and with the equivalent number of parameters, and also achieves promising performance on Pascal VOC and Pascal Context dataset. Our DifNet is trained end-to-end without complex loss functions.

This paper proposes a method for semantic image segmentation in which a single network integrates initial per-pixel class predictions from a DeepLab V2[7]-based network with pairwise affinities derived from learned per-pixel embeddings.  The final predictions of the network are obtained by updating the initial predictions using a series of matrix multiplies by transition matrices computed efficiently from the pairwise affinities.   The main contribution of the work seems to be the use of multiple embeddings, rather than a single one, and the use of a more computationally-efficient inference procedure than some, but not all, prior works.  The authors evaluate the proposed method on the Pascal VOC and Pascal Context datasets, and compare to the Deeplab V2 [7] approach on which the proposed method is based.  Strengths:  - The entire network is trained end-to-end, directly with a cross-entropy loss on the final predictions; in contrast, prior work has relied on more complicated, ad-hoc training strategies, although in many cases still end-to-end.  - DeepLab V2 is a reasonable baseline for comparison.  - The Pascal VOC 2012 and Pascal Context datasets are reasonable choices for evaluation.  - Proposed method is much more computationally efficient than DeepLab V2 with the CRF step included, and is also more computationally efficient than a similar prior work [3].  Weaknesses:  - While the proposed method outperforms certain restricted variants of DeepLab V2 [7], it performs significantly worse than the full DeepLab V2 on both of the two datasets used for comparisons.  - The experimental results show that the proposed method outperforms a restricted variant of DeepLab V2 that does not include the MSC, COCO, and CRF improvements (that are also not used with the proposed method).  However, the proposed method has worse accuracy than the full DeepLab V2 approach, with a particularly large accuracy gap on the Pascal VOC 2012 dataset.  The authors should give results for the proposed method in combination with the MSC, COCO, and CRF improvements.  - The proposed method is very similar to two prior works cited by the authors [3, 5].  The paper should more clearly compare the proposed method to these prior works, and include them for comparison in the tables of experimental results.  - The proposed method seems to have lower accuracy than the two very similar prior works [3] and [5], and also seems to offer no advantage in terms of computational efficiency over [5].  - The paper would benefit from proofreading of the grammar.  Response to author rebuttal: I thank the authors for clarifying the computational complexity of the diffusion step, specifically the fact that it is done with the features downsampled to 64x64; this addresses my confusion regarding the reported computation times.  I appreciate the difficulty of reproducing and comparing against [3] and [5] without source code, and that the that they may also be more difficult to train than the proposed method.  While a more robust method is a valuable contribution even if it does not provider better accuracy or speed, I think a more thorough investigation than is described in the paper would be needed to show that.  Regarding the lack of evaluation with the DeepLab MSC, COCO and CRF steps, while the authors make the valid point that the MSC step and especially the CRF step increase the computational complexity, and that the COCO pretraining adds training data and therefore it may not be fair to compare COCO-pretrained models to those without it, evaluations including these steps are nonetheless important for properly comparing this method to other methods, given the existing similar work.


Model-Agnostic Private Learning
We design differentially private learning algorithms that are agnostic to the learning model assuming access to a limited amount of unlabeled public data. First, we provide a new differentially private algorithm for answering a sequence of m online classification queries (given by a sequence of m unlabeled public feature vectors) based on a private training set. Our algorithm follows the paradigm of subsample-and-aggregate, in which any generic non-private learner is trained on disjoint subsets of the private training set, and then for each classification query, the votes of the resulting classifiers ensemble are aggregated in a differentially private fashion. Our private aggregation is based on a novel combination of the distance-to-instability framework [26], and the sparse-vector technique [15,18]. We show that our algorithm makes a conservative use of the privacy budget. In particular, if the underlying non-private learner yields a classification error of at most α ∈ (0, 1), then our construction answers more queries, by at least a factor of 1/α in some cases, than what is implied by a straightforward application of the advanced composition theorem for differential privacy. Next, we apply the knowledge transfer technique to construct a private learner that outputs a classifier, which can be used to answer an unlimited number of queries. In the PAC model, we analyze our construction and prove upper bounds on the sample complexity for both the realizable and the non-realizable cases. Similar to non-private sample complexity, our bounds are completely characterized by the VC dimension of the concept class.

The paper considers a new differentially private learning setting, that receives a collection of unlabeled public data, on top of the labelled private data of interests and assumes that the two data sets are drawn from the same distribution. The proposed technique allows the use of (non-private) agnostic PAC learners as black boxes oracles, which, when combining with  and adapts to the structure of the data sets.  The idea is summarized below: 1. Do differentially private model-serving in a data-adaptive fashion, through ``sparse vector’’ technique and ``subsample-and-aggregate’’.  This only handles a finite number of classification queries. 2. Reduction: Use the labeled private data and non-private blackbox learner to differentially privately release labels, so the released model is DP w.r.t. the private data since it is just post-processing. It behaves similarly to 1 using the properties of an agnostic PAC learner, but can now handle an unbounded number of classification queries.  The idea is not entirely new but the particular approach with Algorithm 1 and formal theoretical analysis is (See Point 1 in detailed comments).   My only concern is that it is hard to compare the results to the standard non-adaptive DP-learning setting without unlabeled data and their respective upper/lower bounds.  (See Point 2 in my detailed comments)  Other than that, I like the work and I recommend accepting it for NIPS.    Detailed comments:   1. The PATE algorithm used in  - Papernot et al. in "Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data" (ICLR 2017, https://arxiv.org/abs/1610.05755).  is about the same algorithmically, although Papernot et. al. focuses on empirical evaluation.   The differentially private model-serving part is closely related to   - Dwork, C., & Feldman, V. (2018). Privacy-preserving Prediction. arXiv preprint arXiv:1803.10266.  That said, the authors did acknowledge the above works and claim that they are independent.   2. One of the main motivation of the work, according to Line 35-37 is to “characterize … the improvements one can achieve for private learning … ” (with the additional unlabeled data) “compared to the aforementioned setting” (the setting without such data).  I could not find a theorem in the paper that separates the two settings in terms of their sample-complexity. Maybe I missed it somewhere?   Intuitively, there shouldn’t be much difference, especially in the setting when m is about the same as n. Because one can throw away half of the labels in the standard setting so it becomes the setting here with half the data. The only concerning matter is that it is not clear whether the proposed algorithm actually attains the lower bound in the standard setting because the nature of the guarantees are very different, and the current results need to work for all (agnostic) PAC learners.  Is logistic regression is an agnostic PAC learner from the surrogate loss point of view?   Post rebuttal comments:  - I admit having written something misleading in the review and I apologize. Indeed the setting is easier than the standard private learning, where both features and labels need to be protected.   - My comment that the bounds are not compatible still hold, because this paper works with universal learner oracles, while in standard DP learning, the metric of interest is about empirical excess risk (defined by any convex loss, rather than the 0-1 loss here).  - The point about "throwing away a random half of the labels" is in fact about the label-private learning setting. What I would like the authors to compare is the proposed approach and the alternative approach, e.g., that by Chaudhuri and Hsu, which directly analyze the label-privacy for either output perturbation or exponential mechanism.  The advantage of the "model agnostic" approach is not clear to me.  Regardless, this is a much better paper relative to other papers in my stack.  I trust that the authors will do a good revision and clarify these matters in the camera-ready version.


Training DNNs with Hybrid Block Floating Point
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full-precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixedpoint arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5× higher throughput.

This work suggests a new approach for fixed+floating point hybrid. To enable this, use a shared exponent per tensor that allows simple conversion between the formats. They use a fixed point representation for GEMMs to allow only mantissa multiplication while using floating point for all other operations. They also introduce a conversion scheme between intermediate representations of weights and activations. The utility of this approach is demonstrated by simulations and hardware implementation.  Overall this is a nice work, with convincing arguments and experiments. Most noteworthy, I enjoyed seeing an empirical justification for favoring a large exponent, while using a trimmed mantissa. This echoes similar observations by others (e.g BF16 format used by Google). I think that additional comparison to Nervana's Flexpoint format [1] is in place, as it also used an exponent shared scheme, although with different choices for exponent and mantissa sizes.  There are some aspects that I will be happy to see covered: 1) Bandwidth vs computational bound operations - your methods will probably not improve run-time for several operations, it's interesting to see some kind of tear-down for a network such as ResNet50.  E.g - where were the bottlenecks for a floating-point implementation, and for yours (see [2])? 2) Batch norm – You mentioned that “While activation traffic is dwarfed by weight traffic in fully connected, in convolutional layers  … not significant throughput factor”. I would argue that batch-norm is highly bandwidth bound, and that it should be discussed further in your work (most noteworthy, as batch-norm is done completely in fp32 from what I gather). 3) Estimating the cost of conversion operations – you mentioned both finding maximum value in tensor and stochastic rounding to be needed for quantization. I’ll be happy to get an estimate for the cost of these operations (runtime) and their importance in terms of accuracy. 4) A more challenging recurrent task – PTB is good, but I’ll be happy to see a more challenging benchmark such as machine translation to be assured that your methods indeed transfer to other models.  Strengths: - Convincing arguments for a novel numerical representation for DNN training - Good empirical experiments  Weaknesses: - Missing analysis for operations that are not computationally bound (batch-norm) - Estimating the relative cost for needed utility operations (maximum finding, stochatic rounding)  [1] - https://arxiv.org/abs/1711.02213 [2] - https://arxiv.org/abs/1709.08145


A Model for Learned Bloom Filters, and Optimizing by Sandwiching
Recent work has suggested enhancing Bloom filters by using a pre-filter, based on applying machine learning to determine a function that models the data set the Bloom filter is meant to represent. Here we model such learned Bloom filters, with the following outcomes: (1) we clarify what guarantees can and cannot be associated with such a structure; (2) we show how to estimate what size the learning function must obtain in order to obtain improved performance; (3) we provide a simple method, sandwiching, for optimizing learned Bloom filters; and (4) we propose a design and analysis approach for a learned Bloomier filter, based on our modeling approach.

This a purely theoretical paper that is concerned with modeling learned Bloom filters (which are a combination of the statistical filter with the backup classic-Bloom eliminating false negatives arising from the use of the statistical model). In addition to analyzing this method, it proposes a sandwiched version of the learned Bloom-filter (classic-Bloom, statistical-filter, classic-Bloom). It also derives a relationship between the size of the learned model and its associated false positive and negative rates, which entail the same or better OVERALL false positive rate of the learned Bloom filter.  I thank the author(s) for the interesting read. I have found it quite educational. However, I have some mixed feelings about the paper. Below, I talk about these in details. But, here are the few highlights: i) Authors focus primarily on the analysis of the false positive rate at a given bit budget. I think it is an important step. However, as noted by Kraska et al an advantage of the learned filter is that it can use the given bit budget more efficiently. In particular, computing the prediction of the neural network uses easy-to-parallelize operations with localized memory accesses and do not suffer from branch misprediction. Furthermore, if run on GPUs they can use of a much faster GPU memory (matrix multiplication on CPU can be very memory-efficient too). I think this warrants at least a discussion.   ii) Perhaps, I am missing something but Section 3.2 is not very useful, because it does not tell how to choose tau without actually running experiments on the test set.  iii) It looks like authors make somewhat inaccurate claims about properties of the classic Bloom filters.   *PROS*  Paper makes the initial and certainly important step towards analyzing learned Bloom filters.  It provides useful examples, which shed light on behavior/properties of the learned Bloom filters: I find these quite educational.  In particular, it illustrates how easy performance of the Bloom filters can deteriorate when there is a mismatch   It proposes two extensions of the Bloom filters, in particular, a sandwiched Bloom filter.  *CONS*  It feels that the paper was somewhat hastily written (see some comments below). The prose is somewhat hard to understand. In part, it is because of the long sentences (sometimes not very coherent). Sometimes, it is because major findings and observations are buried somewhere. I would encourage you to have some short list of easy-to-grasp high-level overview of the findings. For example, it would be a pity to miss notes in lines 238-242.  The discussion in lines 71-81 makes an assumption that our hash functions operate like random and independent variables. Of course, this is not true in practice, but how much is the difference in realistic settings? Furthermore, this makes you say that you can estimate the false positive rate from any subset of queries. My guess would be that this is true only for good-enough families for hash functions. Although classic Bloom filters should be much less susceptible to distribution changes, I don't think they are total immune to them. In particular, I believe there may be adversarial sequences of keys with much worse FP rate. Discussing these aspects seems to would have been helpful. Last, but not least, as mentioned in the highlights the efficiency of the learned Bloom filters for a given number of bits can be better compared to the classic Bloom filter. In particular, because one lookup may require up to k random memory accesses. From this practical perspective, it is not clear to me if the sandwiched approach always makes sense (as it entails additional memory accesses). In particular, for roughly the same false positive ratio and comparable bit budgets, the original learned Bloom filter may be more efficient than than sandwiched version. In particular, I would like to see theory and/or experiments on this. What if we take the budget for the prefilter in the sandwiched version and give it all to the statistical filter?  *SOME DETAILED COMMENTS*   55:  universe to independently and uniformly to a number   : please check if this is grammatically correct.   63: how? Since you provide an overview anyways, it would be nice to explain that you consider a probability of setting each bit as a random independent variable. Then, not setting it to 1 kn times in a row has such and such probability.   64: Which bound do you use? I guess it's for the binomial with the Chernoff or Hoeffding inequality, but it's never bad to clarify.  67:  what the probability is than an element y will be a false positive -> what the probability is that   67:  Because of this, it is usual to talk ->  because it is tightly concentrated around its expectation, it is usual to talk   109: universe possible query keys -> universe OF possible query keys  122: and 500 -> and OF 500  123: the first half of the line has extra text  120-144: a nice useful illustration  227: it is truly to great to some analysis, but what is missing there is the relationship between \theta and F_n. It doesn't have to be theoretical: some empirical evaluation can be useful too.  238-242: This are really nice points, but it is buried deep in the middle of the paper  254-258: It is not only about improving false positive ratio at a given number of bits. The pre-filtering using a Bloom filter has a non-trivial cost, which includes several random memory accesses. The advantage of having a learned filter, which is implemented by a compact neural net, is that computation over that network is very memory-local and can be easily outsourced to GPU (and kept in a super-fast GPU memory).


Graphical model inference: Sequential Monte Carlo meets deterministic approximations
Approximate inference in probabilistic graphical models (PGMs) can be grouped into deterministic methods and Monte-Carlo-based methods. The former can often provide accurate and rapid inferences, but are typically associated with biases that are hard to quantify. The latter enjoy asymptotic consistency, but can suffer from high computational costs. In this paper we present a way of bridging the gap between deterministic and stochastic inference. Specifically, we suggest an efficient sequential Monte Carlo (SMC) algorithm for PGMs which can leverage the output from deterministic inference methods. While generally applicable, we show explicitly how this can be done with loopy belief propagation, expectation propagation, and Laplace approximations. The resulting algorithm can be viewed as a post-correction of the biases associated with these methods and, indeed, numerical results show clear improvements over the baseline deterministic methods as well as over "plain" SMC.

Summary and Assessment: ------------------------  This paper strives to improve Sequential Monte Carlo (SMC) sampling on probabilistic graphical models through the usage of twisted targets. More specifically, rather employ a "myopic" sequence of target distributions consisting of gradually introducing the factors and variables in the overall target (according to some ordering criteria) a method is devised by which the future can be conditionally approximated and taken into account. The idea is to devise a target that more closely approximates the true marginal distribution (\pi( x_1,...x_t) of \pi at step t rather than that resulting from dropping all future interactions. Proposition 1 presents the ideal but infeasible choice of twisting function. In effect, equation (6) defines a conditional partition function, and so approximating it with a deterministic method seems sensible. The authors present loopy BP, EP, and Laplace approximation approaches to achieve this. Improvements over myopic SMC are demonstrated in several regimes.   This is a good paper with nice ideas; I am encouraged to see ongoing research into improved sampling for PGMs. The paper is very well-written and clearly motivates the problem and contributions. Inference in PGMs remains an important problem today, and the authors present a natural and intuitively appealing enhancement to sampling techniques on these models. I believe this work should be accepted to NIPS.  The results and statements are technically correct to my knowledge.  Questions and Minor Issues: --------------------------  - Can more be said about the computational burden of the variational fitting? On line 232, the cost of BP is described as "quite small" which may well be true for 16x16 lattices, but the issue is how does such effort scale with problem size. Nonetheless I can readily believe that in many situations, the extra cost will be more than compensated for in the overall efficiency compared to myopic SMC.  - This is my most "serious" concern: is there any reason the authors did not include comparisons to SMC under traditional temperature-based annealing? In such a scheme, of course, the overall distribution is always considered, though at different "smoothness" due to the introduction of high-temperature target distributions. It is my experience that for "complex" energy landscapes, it is often quite difficult to do better than a temperature-based sequence. Quite likely, as the authors' results suggest, the cases presently considered are not such "rough" problems, as hinted at by the success of the variational approximations, but in those cases thermal annealing also tends to work quite well. I do suggest that in future work, the authors investigate this if they have not already done so.  - A suggestion regarding related work: a relatively early application of SMC to PGMs was Hamze and de Freitas (NIPS '05), where edges (potentials) were sequentially introduced to form the target sequence.  - A natural question one may ask, given that the method does something along the lines of a "lookahead", and that equation (6) specifies a conditional partition function, is how well a recursive application of the naive SMC method to approximate the optimal twisted target would fare. More specifically, (6) could be approximated with the myopic SMC (at step t, over variables t+1 to T), but with simulation parameters setting up a fast "pilot" run to get a rough estimate; it would be interesting to see how this does against the variational approximations. Have the authors considered such an approach?


Escaping Saddle Points in Constrained Optimization
In this paper, we study the problem of escaping from saddle points in smooth nonconvex optimization problems subject to a convex set C. We propose a generic framework that yields convergence to a second-order stationary point of the problem, if the convex set C is simple for a quadratic objective function. Specifically, our results hold if one can find a ⇢-approximate solution of a quadratic program subject to C in polynomial time, where ⇢ < 1 is a positive constant that depends on the structure of the set C. Under this condition, we show that the sequence of iterates generated by the proposed framework reaches an (✏, )-second order stationary point (SOSP) in at most O(max{✏ 2 , ⇢ 3 3 }) iterations. We further characterize the overall complexity of reaching an SOSP when the convex set C can be written as a set of quadratic constraints and the objective function Hessian has a specific structure over the convex set C. Finally, we extend our results to the stochastic setting and characterize the number of stochastic gradient and Hessian evaluations to reach an (✏, )-SOSP.

In this submission, the authors propose an algorithm for nonconvex optimization under convex constraints. The method aims at escaping saddle points and converging to second-order stationary points. The main idea of the submission lies in exploiting the structure of the convex constraint set so as to find an approximate solution to quadratic programs over this set in polynomial time. By combining this oracle with a first-order iteration such as conditional gradient or projected gradient, the authors are able to guarantee that their algorithm will converge towards an $(\epsilon,\gamma)$-Second-Order Stationary Point (SOSP), i. e. a point at which first and second-order necessary conditions are approximately satisfied, respectively to $\epsilon$ and $\gamma$ tolerances.  In addition to addressing a topic of increasing interest in the community of NIPS (escaping saddle points in nonconvex optimization), this submission also touches complexity aspects that are of general interest in theoretical computer science (NP-hardness and approximation algorithms).  The authors clearly introduce the problem and their objectives, while positioning it within the existing literature. My main comment about the introductory section is that this submission claims that it provides ``the first finite-time complexity analysis for reaching an SOSP" of the considered problem. I believe this is not entirely true, considering the following work:  C. Cartis, N. I. M. Gould and Ph. L. Toint, ``Second-Order Optimality and Beyond: Characterization and Evaluation Complexity in Convexly Constrained Nonlinear Optimization", Foundations of Computational Mathematics,  DOI 10.1007/s10208-017-9363-y.  In this reference, the authors derive a bound in $\epsilon^{-3}$, in terms of outer iterations and function/derivatives evaluations, for obtaining a point at which approximate second-order conditions are satisfied. Although their optimality measure differs from that of this submission, I believe it is clear that their method also escapes saddle points and is thus worth mentioning (as mentioned by the authors regarding other references, the cost of solving the subproblem is not explicitly taken into account). Their optimality measure only involves one tolerance, yet one can view their definition of first-order optimality as similar to that of the submission, and I would welcome the authors' comments on this reference.  I would have expected Definitions 2 and 3 to enforce feasibility of the point in addition to the other optimality conditions. In the present submission, this is not a critical issue as all considered iterates are computed so as to be feasible, however I believe adding ``feasible" in these definitions would make more sense, because it does not seem that infeasible points are desirable in the context of this submission. I would like to have the authors' view on this point.  The presentation of the main algorithm and result in Section 3 is in my opinion well conducted. Even though it is clear from the algorithmic process, the authors could have insisted more on the fact that the method can have cheap first-order iterations, which is one of the strengths of this framework. The description of the first-order steps and algorithms in Section 4 is extremely clear.  Section 5 provides an example of the second phase of the algorithm (second-order step) when the constraint set is defined as an intersection of ellipsoids. Using a result from approximation algorithms, the authors show that in that case, it is possible to obtain a solution sufficiently close to the global optimum, thereby enabling the derivation of a complexity bound. Another example for linear constraints is provided in appendix. One might be concerned with the applicability of such results, especially given their dependencies on the problem dimension. Still, the argument appears valid, and the considered cases are among the most commonly encountered.  The last technical section of the submission, Section 6, is concerned with extending the previous results to the case where the gradient and Hessian of the objective are only accessed through stochastic estimates. Using batch gradients and Hessians with appropriate batch sizes, it is possible to obtain complexity guarantees for reaching approximate second-order stationary points with high probability. This extension and the associated algorithmic modifications are very interesting: in particular, the relaxation of the constraint $d_t^\top (u-x_t)=0$ is a key feature of the stochastic method, which is briefly but clearly discussed by the authors. It seems that the condition on Line 3 of Algorithm 2 is also a key difference, about which a comment could have been made. In addition, the first-order-type step is computed at every iteration, probably as a consequence of the stochasticity: this is a key difference in the per-iteration cost of the method, even though it does not impact the complexity analysis. I acknowledge that the authors could not add an extensive discussion due to space constraints, but part of it could have been added as supplementary material.  In a nutshell, I believe the results of the submission to be correct, clearly introduced and motivated. Although I regret the lack of numerical experiments that would have illustrated the applicability of the proposed framework, it is my opinion that this submission can still have an impact on the community, and stimulate further work on algorithms of this form.  Therefore, I will give this submission an overall score of 8.  Additional comments:  a) There are a few minor typos, e.g. "propsoed" line 43.  b) The series of equations on line 523 page 15 is an interesting derivation, but the first lines might need clarification. More precisely, I would like the authors to comment upon the two expectation operators that appear in the first three equalities, and the argument for going from the third equality to the fourth one.  Edit following author feedback:  I thank the authors for providing detailed answers to my comments, and I trust them to incorporate revisions as stated. I agree with the other reviewers' comments regarding the ability of the proposed scheme to escape strict saddle points, but I am confident that this can be addressed by the authors. As a result, I maintain my initial assessment on the submission.


An intriguing failing of convolutional neural networks and the CoordConv solution
Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x, y) Cartesian space and coordinates in one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious. We call this solution CoordConv, which works by giving convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv allows networks to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. CoordConv solves the coordinate transform problem with perfect generalization and 150 times faster with 10-100 times fewer parameters than convolution. This stark contrast raises the question: to what extent has this inability of convolution persisted insidiously inside other tasks, subtly hampering performance from within? A complete answer to this question will require further investigation, but we show preliminary evidence that swapping convolution for CoordConv can improve models on a diverse set of tasks. Using CoordConv in a GAN produced less mode collapse as the transform between high-level spatial latents and pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST detection showed 24% better IOU when using CoordConv, and in the Reinforcement Learning (RL) domain agents playing Atari games benefit significantly from the use of CoordConv layers.

Update following author response  Thank you for addressing many of the concerns. Overall, all my previous concerns about the quality, clarity and significance are alleviated. However I am not very certain about the originality still.  ------------------------  - summary of the main ideas  It was difficult to provide a concise summary of the main ideas because there is a lot going on in this paper. Note that various models were used for experimentation and that model specifications are given in the supplementary material. It was also difficult to keep track of which kind of convolutional neural networks (CNNs) were used for which subproblem.  First, this paper shows instances where CNNs fail at particular subsets of the coordinate transform problem. For "normal" CNNs, the paper shows:   1. that a fully convolutional model fails to solve the supervised coordinate classification problem and the supervised rendering problem.  2. that a convolutional model with max-pooling and fully connected layers fails to solve the supervised coordinate regression problem, specifically the quadrant split. However the respective model generalizes well for the supervised coordinate regression problem with the uniform split. 3. that a fully convolutional model achieves limited generalization on the supervised coordinate regression problem with quadrant split. The respective model fails on the supervised coordinate regression problem with uniform split.   Second, a solution called CoordConv is proposed to solve this problem, generalized as the coordinate transform problem. With CoordConv, the input to the CNN is augmented with 2 extra channels representing the x and y Cartesian coordinates respectively. This is the only modification needed to implement CoordConv. They show that using CoordConv, all the above mentioned toy problems are solved with extremely high accuracy, requiring far less training time and utilizing smaller CNNs (less model parameters).   Third, to show that the CoordConv solution can increase performance on other problems, and not just this toy problem, a comparison is made between "normal" CNNs and CoordConv CNNs on the following problems: - Unsupervised Density Learning with GANs - Image classification with ResNet-50 - Deep Reinforcement Learning (RL) on a selection of Atari games with the A2C network   - quality -- strengths .Performance on the toy problems was demonstrated very well and the results were impressive. .Figure 8: (a) I was surprised to see the great improvement in performance because MsPacman is notoriously difficult.  .In the RL experiment and the image classification task, significance testing was performed. -- weaknesses .In lines 22 and 72 it is mentioned that for the ImageNet classification with ResNet-50 the performance increase is significant, with p=0.051. This is simply a false claim, which you seem to realize as well, since you write in line 224 that this result is "nearly statistically significant".  .Figure 7: The results of the GAN and CoordConv GAN are ambiguous. For example, I see no difference between the two models in (a) and in (c) I can argue that the CoordConv is the one suffering from mode collapse, since the "normal" GAN shows a more diverse sampling. For (d) I don't know how to interpret this. .No quantifiable measure for mode collapse is used. It would have been much better if an objective measure for mode collapse had been used, such as entropy.  .Figure 8: (b) it is highly questionable that this graph is cut off at timestep 7, exactly when the overlap in performance happens. This result seems cherry-picked. Especially since (a) and (c) go up to timestep 8. Please explain why this was done, ideally provide a graph that goes to timestep 8. Also, why is there no table of all the Atari games together with their results? .No theoretical justification was given whatsoever for the behaviors exhibited by the various neural networks. The rigor of the experiments outside of the toy context was not satisfactory to a NIPS standard. .The role of max-pooling for achieving translation invariance is not addressed. For example, no explanation is given for the differences in performance between (2) and (3) in my summary above. To me it is clear that max-pooling will be problematic for tasks that require translation dependence, since max-pooling increases translation invariance. .In line 162 the paper claims to verify that "performance on this task does in fact predict performance on larger problems". However, no such verification is presented. .The reasoning for the demonstration on the RL task and the image classification task is not given. I am left with the question of "why would these specific tasks benefit from CoordConv?". To me it seems that a location specific task would benefit from CoordConv, however with the image classification I don't see the usefulness of CoordConv because ImageNet classification task is translation invariant.   - clarity -- strengths .Mostly very good visuals, clearly representing the benefit of CoordConv on the toy problems. .Paper was well structured -- weaknesses .It was difficult to keep track of which kind of convolutional neural networks (CNNs) were used for which subproblem. A table showing an overview of the various models would have been very useful. .The paper does not provide enough information for replication. Even though it is mentioned how the grid search for parameters is performed, it is not clear which final networks were used to obtain the figures. .Figure 7 did not give a clear overview of the differences between the models.  .It is not clear to me how to implement the CoordConv solution in the supervised coordinate classification problem and the supervised rendering problem. In Figure S1, what does "32-32-64-64-1" mean?  - originality -- strengths .This is the first time that I have seen such an approach to expand the capabilities of a CNN towards translation dependence.  -- weaknesses .I am not entirely sure how original this contribution is, given that there is another paper which uses a similar approach of channel coordinate augmentation in their pipeline to perform image inpainting, see Figure 2 in the supplementary material: Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. "Deep image prior." arXiv preprint arXiv:1711.10925 (2017). Supplementary material: https://box.skoltech.ru/index.php/s/ib52BOoV58ztuPM#pdfviewer However said paper does not make this the main focus of their work and it is also not clear to me if the "meshgrid" channels are directly added to the input.  - significance -- strengths .The failing of traditional CNNs (with max pooling or fully convolutional) has been adequately demonstrated on the toy problems. It is also clear that the CoordConv solution solves these problems. -- weaknesses .However, it is not clear to me that this solution is useful beyond a toy context.


Multi-Task Learning as Multi-Objective Optimization
In multi-task learning, multiple tasks are solved jointly, sharing inductive bias between them. Multi-task learning is inherently a multi-objective problem because different tasks may conflict, necessitating a trade-off. A common compromise is to optimize a proxy objective that minimizes a weighted linear combination of pertask losses. However, this workaround is only valid when the tasks do not compete, which is rarely the case. In this paper, we explicitly cast multi-task learning as multi-objective optimization, with the overall objective of finding a Pareto optimal solution. To this end, we use algorithms developed in the gradient-based multiobjective optimization literature. These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks. We therefore propose an upper bound for the multi-objective loss and show that it can be optimized efficiently. We further prove that optimizing this upper bound yields a Pareto optimal solution under realistic assumptions. We apply our method to a variety of multi-task deep learning problems including digit classification, scene understanding (joint semantic segmentation, instance segmentation, and depth estimation), and multilabel classification. Our method produces higher-performing models than recent multi-task learning formulations or per-task training.

Overall summary of the paper:  This paper proposed a multi-task learning algorithm from multi-objective optimization perspective and the authors provided an approximation algorithm, which could accelerate the training process. The authors claim that existing MTL algorithms used linear combinations (uniform weight) of the loss from different tasks, which is hard to achieve the Pareto optimality. Unlike the uniform weight strategy, the authors use the MGDA algorithm to solve the optimal weight, which would increase the performance for all the tasks to achieve the Pareto optimality. Moreover, when solving the sub-problem for shared parameters, the author gave an upper bound of the loss function, and this upper bound optimization only requires one-time back propagation regardless of the number of tasks. The results show that the approximation strategy not only can accelerate the training process but also improve the performance.  Strengths: 1. The proposed method can solve the optimal weight for the loss of different tasks. 2. The approximation strategy can efficiently accelerate the training process. 3. The proposed algorithm is easy to implement.  Weaknesses: 1. The motivation is not convincing enough. The decrease of training loss does not mean the generalization performance is also good. Pareto optimality is for the training loss decrease, which has no guarantee that it will also improve the generalization performance. 2. The optimization treatment is not novel.  3. The approximation provides a good and easy way to accelerate the algorithm, which is the spotlight of this paper. However, in the experiment, the approximation one is even better than the original one in prediction, which does not make sense. Although the authors provide some qualitative reasons at the end of the paper, it is hard to convince me why this approximation can even improve the prediction result. I noticed that in Table 2, the training time for without approximation is longer than 10 days. Is the baseline without approximation carefully tuned?  4. The author does not include the uniform scaling + approximation strategy. I am wondering how this baseline perform compared to other methods in the paper. This could help readers to understand which part is really effective.


Latent Alignment and Variational Attention
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property makes it difficult to compare attention to other alignment approaches, to compose it with probabilistic models, and to perform posterior inference conditioned on observed data. A related latent approach, hard attention, fixes these issues, but is generally harder to train and less accurate. This work considers variational attention networks, alternatives to soft and hard attention for learning latent variable alignment models, with tighter approximation bounds based on amortized variational inference. We further propose methods for reducing the variance of gradients to make these approaches computationally feasible. Experiments show that for machine translation and visual question answering, inefficient exact latent variable models outperform standard neural attention, but these gains go away when using hard attention based training. On the other hand, variational attention retains most of the performance gain but with training speed comparable to neural attention.

Update based on author rebuttal: I believe the authors have addressed the main criticisms of this paper (not clear how it's different from prior work) and have also provided additional experiments. I've raised my score accordingly.  This paper focuses on using variational inference to train models with a "latent" (stochastic) attention mechanism. They consider attention as a categorical or dirichlet random variable, and explore using posterior inference on alignment decisions. They test various approaches on NMT and visual question answering datasets.  Overall, this was an interesting paper exploring some unusual methods for training models with stochastic/discrete attention. As discussed by the paper, the main reasons to explore this approach are 1) the possibility of improved performance and 2) better interpretability.  The paper achieves 1) to a reasonable degree - in both NMT and VQA the latent attention-based models are able to improve over standard softmax attention, though to a limited degree. It's not possible to compare the NMT results to most results in the literature as the experiments were done on IWSLT instead of WMT, but this paper is more structured as an exploration of different ways to train latent attention models rather than an attempt at state-of-the-art. I don't think the paper suitably addresses 2). Only Figure 2 really explores whether the alignments are more interpretable, and only to a limited degree.  In prior work (Xu et al.) it was shown that while hard attention can produce better results, the alignments can actually be *less* interpretable. I would urge the authors to either more rigorously back up their claim that hard attention is more interpretable, or temper their claims somewhat. Finally, in various places in the paper it is pointed out that Ba et al. and Lawson et al. have both previously explored variational inference approaches for training attention-based models, but no direct comparison to those techniques is presented. If any of the methods explored subsume those approaches and I just missed this connection, then I would suggest making the connection stronger and more broadly providing a better comparison of the approaches and results in this paper to those in prior work.  Some minor comments:  - "While more principled..." Is it really more principled (as if there's a definition for "more principled")? I find this statement somewhat objectionable. There are many neural network mechanisms which improve performance or are genuinely useful for reasons separate from whether they can be treated as a latent variable. I think the primary reason people use attention is because it's helpful (probably because of gradient dynamics or otherwise), and it just so happens to be interpretable too. People also use model weights, activations, etc. for interpretability reasons but this doesn't imply it would be more principled (or helpful) to treat them as latent variables.  - "may improve system" -> "may improve systems"  - "Our method uses uses different" -> "Our method uses different"  - "f(x, z) could use z to pick from x" Wouldn't it make more sense to say "f(x, z) could use z to pick from X"?  - "this may does prevent" -> "may prevent"  - "domain structure to for the" -> "domain structure for the"


Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search
We present a learning-based approach to computing solutions for certain NPhard problems. Our approach combines deep learning techniques with useful algorithmic elements from classic heuristics. The central component is a graph convolutional network that is trained to estimate the likelihood, for each vertex in a graph, of whether this vertex is part of the optimal solution. The network is designed and trained to synthesize a diverse set of solutions, which enables rapid exploration of the solution space via tree search. The presented approach is evaluated on four canonical NP-hard problems and five datasets, which include benchmark satisfiability problems and real social network graphs with up to a hundred thousand nodes. Experimental results demonstrate that the presented approach substantially outperforms recent deep learning work, and performs on par with highly optimized state-of-the-art heuristic solvers for some NP-hard problems. Experiments indicate that our approach generalizes across datasets, and scales to graphs that are orders of magnitude larger than those used during training.
of solutions, which enables the network to explicitly disambiguate different modes in the solution space. This trained GCN is used to guide a parallelized tree search procedure that rapidly generates a large number of candidate solutions, one of which is chosen after subsequent refinement.
We apply the presented approach to four canonical NP-hard problems: Satisfiability (SAT), Maximal Independent Set (MIS), Minimum Vertex Cover (MVC), and Maximal Clique (MC). The approach is evaluated on two SAT benchmarks, an MC benchmark, real-world citation network graphs, and social network graphs with up to one hundred thousand nodes from the Stanford Large Network Dataset Collection. The experiments indicate that our approach substantially outperforms recent state-of-the-art (SOTA) deep learning work. For example, on the SATLIB benchmark, our approach solves all of the problems in the test set, while a recent method based on reinforcement learning does not solve any. The experiments also indicate that our approach performs on par with or better than highly-optimized contemporary solvers based on traditional heuristic methods. Furthermore, the experiments indicate that the presented approach generalizes across datasets and scales to graphs that are orders of magnitude larger than those used during training.

This paper proposes a method combining graph neural networks and guided tree search to tackle combinatorial optimization problems. The authors focus on the Maximum Independent Set (MIS), the Minimum Vertex Cover (MVC), Maximal Clique (MC) and Satisfiability which are all reduced to MIS. Given a MIS instance represented by a graph, the method consists in using the predictions of a graph convolutional neural (GCN) network, which is trained with binary cross-entropy to predict whether a vertex is in the maximum independent set, as input for a greedy search algorithm. The greedy search algorithm considers vertices in descending order of the GCN outputs, adding them to the independent set if doing so doesn't violate the independence assumption. Once this isn't possible, the process is repeated on the residual graph until termination. The method is further augmented with diversity, allowing the GCN to output different probability maps, and a tree search (partial solutions are added to a queue from which they will be randomly sampled to be further constructed). They train on a MIS graphs obtained from satisfiable SAT instances from the SATLIB benchmark for which solutions are available. They test the approach on instances coming from synthetic datasets and real-world graphs (such as citation networks or social networks). The authors compare their approach to the S2V-DQN baseline (from Learning Combinatorial Algorithms over Graph), the SOTA ILP solver Gurobi and the SMT solver Z3. Interestingly, the approach transfers well to different data distributions, larger instances and other problems. The proposed approach outperforms S2V-DQN and sometimes outperforms Z3 or Gurobi (which are hard baselines - assuming they're applied on the right problem).  The paper is clearly written and proposes a novel approach for tackling combinatorial optimization (a fundamental field in computer science) using machine learning. Some results are particularly interesting such as the fact that the approach transfers that well to other problems and larger instances or the fact that using the diversity loss provides such an improvement.  However, I have several issues with the current state of this draft (in no particular order). I am of course willing to change my low rating if these issues are addressed.  - Although the proposed method clearly improves over Classic+GR+LS and in spite of the ablation study in Table 4, I think that a random baseline (ie Random+GR+LS) is necessary to evaluate the usefulness of the neural netwrok in the approach. That random baseline should be given the same time budget and should be similar to the proposed approach except for the fact that it wouldn't query the network (ie use a random order).  - Supervised learning is presented as an advantage of the method on the premise that Reinforcement learning is a particularly challenging optimization problem and that supervised learning benefits from well-understood and reliable solvers. I believe that relying on labeled solutions is a weakness (albeit small) of the approach since it requires having solved (even approximately) NP-Hard problems. This is echoed in Neural Combinatorial Optimization with Reinforcement Learning (NCO), where the authors show that RL training clearly outperforms Supervised Learning training without requiring the need for costly solutions.  - The S2V-DQN baseline is quite weak in my opinion: *) I take exception that S2V-DQN outperforms prior deep learning approaches such as NCO: the results that they report when reimplementing NCO are worse than those initially reported in NCO paper, they used a moving average baseline instead of a critic, they don't use the fact that NCO can sample many solutions as one etc... *) As correctly pointed out by the authors, S2V-DQN only outputs one solution while their approach considers multiple solutions. In comparison, NCO and Attention solves your TSP, approximately (which uses a Transformer-Pointer Network instead of an RNN Pointer Network) can output many solutions (when sampling) making them better baselines to the proposed approach. *) Attention solves your TSP, approximately does outperform prior learning-based approaches (and actually shows that the NCO approach outperforms S2V-DQN) This contrasts to the overall tone of the paper that their method clearly surpasses recent work ("Our algorithm convincingly outperforms recent work" in the conclusion)  - The authors should also cite recent machine learning / combinatorial optimization work such as NeuroSAT, Attention solves your TSP,  Learning the Multiple Traveling Salesmen Problem with Permutation Invariant Pooling Networks, etc.  =================================================  ) Random+GR+LS: The 12.7% performance of this baseline outlines how important it is to consider multiple solutions.  2) Relying on supervised data is a weakness: I misunderstood that the labels relied on solvers. I agree that this is not an issue if the labels are generated without a solver and the authors have shown strong generalization.  3)  The S2V-DQN baseline is quite weak. Out of the considered learning based approaches (ie NCO, ASTSP, the authors' method and S2V-DQN), S2V-DQN is the only method that considers a single solution.  NCO and ASTSP can easily consider multiple solutions (by sampling) but it looks like the authors use the greedy version ASTSP only (this is hard to tell from the rebuttal but given that ASTSP solves 0% while random search solves 12.7% this seems to be the case).  4) I also invite the authors to reconsider the following statements which are incorrect / misleading. (a) "Most recently, Dai et al. [10] used reinforcement learning to train a deep Q-network (DQN) to incrementally construct solutions to graph-based NP-hard problems, and showed that this approach outperforms prior learning-based techniques." - For tasks where the methods are comparable (and compared), Dai et al report worse results when reimplementing NCO than those initially reported in NCO paper. Careful reading indicates that they used a moving average baseline instead of a critic, they don't use the fact that NCO can sample many solutions as one, etc... This is also echoed in a figure from the independent ASTSP paper which present NCO and ASTSP as outperforming Dai et al.  (b) "First, we do not use reinforcement learning, which is known as a particularly challenging optimization problem. Rather, we show that very strong performance and generalization can be achieved with supervised learning, which benefits from well-understood and reliable solvers." - Previous work in combinatorial optimization has shown that RL is better suited than supervised learning (at least for pointer network models). - Reliance on solvers is a weakness in general.  (c) (rebuttal) "RL-based approaches consider problems up to 100 variables". This is not true (although the authors have indeed tackled larger problems that prior work). It also not clear why this statement is relevant to the comparison of RL vs supervised learning: I think it has more to do with the types of models being used rather than the training method.  In summary, I have a significant presentation issue with this paper.  I agree that % of solved instances is an important practical metric but it obviously greatly favors methods that consider multiple solutions. The paper does a poor job at outlining this fact and compares only to learning based methods that only consider a single solution (ie S2V-DQN). The fact that all other learning based are outperformed by the random search further demonstrates my point. Methods tend to be much closer when looking at the average independent set size metric. (The authors also didn't report this metric for ASTSP or Random+GR+LS in the rebuttal).  Relying on search (ie trying multiple solutions) to get good results is not a weakness of the paper in itself and I actually think it's a promising direction to combine both classic search methods with machine learning. However comparing to learning-based methods that only consider a single solution when considering to a metric that favors multiple solutions can be misleading . Especially that these methods can easily be augmented by search (ie sampling in NCO and ASTSP). The other incorrect statements about related work (mistakenly saying that the RL-based approaches consider up to 100 variables, etc.) also don't play in favor of the work.  If this wasn't for these issues, this would be a strong paper. I appreciate the thorough response of the authors and I have updated my score to 4.


Entropy and mutual information in models of deep neural networks
We examine a class of stochastic deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics methods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method. (iii) We propose an experiment framework with generative models of synthetic datasets, on which we train deep neural networks with a weight constraint designed so that the assumption in (i) is verified during learning. We study the behavior of entropies and mutual informations throughout learning and conclude that, in the proposed setting, the relationship between compression and generalization remains elusive.
The successes of deep learning methods have spurred efforts towards quantitative modeling of the performance of deep neural networks. In particular, an information-theoretic approach linking generalization capabilities to compression has been receiving increasing interest. The intuition behind the study of mutual informations in latent variable models dates back to the information bottleneck (IB) theory of [1]. Although recently reformulated in the context of deep learning [2], verifying its relevance in practice requires the computation of mutual informations for high-dimensional variables, a notoriously hard problem. Thus, pioneering works in this direction focused either on small network models with discrete (continuous, eventually binned) activations [3], or on linear networks [4,5].
In the present paper we follow a different direction, and build on recent results from statistical physics [6,7] and information theory [8,9] to propose, in Section 1, a formula to compute informationtheoretic quantities for a class of deep neural network models. The models we approach, described in Section 2, are non-linear feed-forward neural networks trained on synthetic datasets with constrained weights. Such networks capture some of the key properties of the deep learning setting that are usually difficult to include in tractable frameworks: non-linearities, arbitrary large width and depth, and correlations in the input data. We demonstrate the proposed method in a series of numerical experiments in Section 3. First observations suggest a rather complex picture, where the role of compression in the generalization ability of deep neural networks is yet to be elucidated.
The successes of deep learning methods have spurred efforts towards quantitative modeling of the performance of deep neural networks. In particular, an information-theoretic approach linking generalization capabilities to compression has been receiving increasing interest. The intuition behind the study of mutual informations in latent variable models dates back to the information bottleneck (IB) theory of [1]. Although recently reformulated in the context of deep learning [2], verifying its relevance in practice requires the computation of mutual informations for high-dimensional variables, a notoriously hard problem. Thus, pioneering works in this direction focused either on small network models with discrete (continuous, eventually binned) activations [3], or on linear networks [4,5].
In the present paper we follow a different direction, and build on recent results from statistical physics [6,7] and information theory [8,9] to propose, in Section 1, a formula to compute informationtheoretic quantities for a class of deep neural network models. The models we approach, described in Section 2, are non-linear feed-forward neural networks trained on synthetic datasets with constrained weights. Such networks capture some of the key properties of the deep learning setting that are usually difficult to include in tractable frameworks: non-linearities, arbitrary large width and depth, and correlations in the input data. We demonstrate the proposed method in a series of numerical experiments in Section 3. First observations suggest a rather complex picture, where the role of compression in the generalization ability of deep neural networks is yet to be elucidated.

Contributions of the paper:  The authors consider a stylized statistical model for data that respects neural network architecture, i.e. a Markov structure of the type T_\ell = \varphi(W_\ell*T_{\ell-1}, \xi_\ell ) where T_0 = X is the input, T_L = y is the output label, $W_\ell$ are random, independent weight matrices, \varphi is a nonlinearity applied elementwise on its first argument, possibly using  external randomness \xi_\ell. For data generated from this specific model, they make the following contributions.   1. They show that under this stylized model, one can obtain a simple formula for the (normalized i.e. per unit) entropy  H(T_\ell)/n and mutual information I(T_\ell; X)/n between the input data and each successive layer, in the high-dimensional limit. This formula is, in general, derived using the non-rigorous  replica method from statistical physics.   2. For a two-layer network and a class of nonlinearities, they rigorously establish this formula, using a technique of `adaptive' interpolation.   3. Based on this model, they provide some experimental results for two synthetic scenarios: the first based on the teacher-student model,  and the second for generative models, such as variational autoencoders. The experimental results are multi-faceted and include a comparison with entropy/mutual information estimators, validation of the replica formula, and some applications to the recent information bottleneck proposal of Tishby et al.    Summary:  The paper is a solid contribution, and I would argue that it is a clear accept. It would make a good addition to the NIPS program  and is of potential interest to communities possibly unfamiliar with the approaches in the paper.   Comments for the authors:  There are multiple aims that you have in the experiments section: which point to interesting applications for your results, like comparing entropy estimators, validating the bottleneck theory, potential training innovations. Further, some of your experiments aim to validate the replica formula where it is not known to hold,  e.g. during training (thanks to dependence). It would be good to add an experiments summary that clarifies these aims and points the readers to the relevant sections. You might have to move some of the  discussion due to space constraint.    I have read the author response. My scores are unchanged, I look forward to the final version of the article.


Collaborative Learning for Deep Neural Networks
We introduce collaborative learning in which multiple classifier heads of the same network are simultaneously trained on the same training data to improve generalization and robustness to label noise with no extra inference cost. It acquires the strengths from auxiliary training, multi-task learning and knowledge distillation. There are two important mechanisms involved in collaborative learning. First, the consensus of multiple views from different classifier heads on the same example provides supplementary information as well as regularization to each classifier, thereby improving generalization. Second, intermediate-level representation (ILR) sharing with backpropagation rescaling aggregates the gradient flows from all heads, which not only reduces training computational complexity, but also facilitates supervision to the shared layers. The empirical results on CIFAR and ImageNet datasets demonstrate that deep neural networks learned as a group in a collaborative way significantly reduce the generalization error and increase the robustness to label noise.

The paper introduces collaborative learning, a modification to architectures that adds additional heads to the classifier in turn enabling better generalization without added inference costs.   Strengths: - The topic of study is of great relevance and significance. While ensembling of networks boosts performance, the added inference cost is typically not warranted; having a way to distill ensemble-like performance in a single model is impactful.  - The authors present results on CIFAR and ImageNet datasets which show how the proposed heuristics improve performance. The ablation experiments are useful and add to the reader's understanding of the changes.   Weaknesses:  - The idea of using heads (or branches) in deep networks is not entirely novel and the authors do not describe or compare against past work such as mixture of experts and mixture of softmax (ICLR 2018). The authors should also investigate the Born Again Networks (ICML 2018) work that uses self-distillation to obtain improved results. It would also be beneficial if the authors described auxiliary training in the related work since they explain their approach using that as foundation (Section 3.1).   - The proposed changes can be explained better. For instance, it is not clear how the hierarchy should be defined for a general classifier and which of the heads is retained during inference.    - What baseline is chosen in Figure 3? Is it also trained on a combination of hard and soft targets?   - The approach presented is quite heuristic it would be desirable if the authors could discuss some theoretical grounding for the proposed ideas. This is especially true for the section on "backpropagation rescaling". Did the authors try using a function that splits the outputs equally for all heads (i.e., I(x) = x/H, I'(x) = 1/H)? The argument for not using sum_h Lh is also not convincing. Why not modulate eta accordingly?  - The paper would benefit from careful editing; there are several typos such as "foucus", "cross entroy", and awkward or colloquial phrases such as "requires to design different classifier heads", "ILR sharing with backpropagation rescaling well aggregates the gradient flow", "confusion often happens to network training", "put on top of individual learning", "SGD has nicer convergence".   Post-rebuttal:  Raising my score to a 6 in response to the rebuttal. I do wish to point out that: - To my understanding, BANs don't ensemble. They only self-distill.  - Is using Lhard the right baseline for Q3?