∆-encoder: an effective sample synthesis method for few-shot object recognition
Learning to classify new categories based on just one or a few examples is a long-standing challenge in modern computer vision. In this work, we propose a simple yet effective method for few-shot (and one-shot) object recognition. Our approach is based on a modified auto-encoder, denoted ∆-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it. The synthesized samples are then used to train a classifier. The proposed approach learns to both extract transferable intra-class deformations, or "deltas", between same-class pairs of training examples, and to apply those deltas to the few provided examples of a novel class (unseen during training) in order to efficiently synthesize samples from that new class. The proposed method improves the state-of-the-art of one-shot object-recognition and performs comparably in the few-shot case.
* The authors have contributed equally to this work Corresponding author: Leonid Karlinsky (leonidka@il.ibm.com) Figure 1: Visualization of two-way one-shot classification trained on synthesized examples. Correctly classified images are framed in magenta (Golden retriever) and yellow (African wild dog). The only two images seen at training time and used for sample synthesis are framed in blue. Note the non-trivial relative arrangement of examples belonging to different classes handled successfully by our approach. The figure is plotted using t-SNE applied to VGG features.

*** After rebuttal: Thank you for the additional experiment, this satisfies my curiosity and convinces me more of the inability of a linear offset to achieve similar results. It is also encouraging that the reported results compare favourably to the recent paper by Gidaris et al. Finally, the provided explanation fully clarify my understanding of the algorithm, and I feel it's a nice and simple idea. I've raised my score to 7 as a result of all the above.  One thing that would be important to clarify is that the feature extractor is trained exclusively on the meta-training set of each dataset considered. Another reviewer misunderstood this fact, and I remember also misunderstanding this during my first pass of the paper. I think the confusion might be due to using the word 'pretrained' which may be interpreted as reusing a feature extractor that was trained on another dataset(s), instead of merely describing a two-stage training phase where first the feature extractor is trained, and then held fixed while the encoder / decoder are trained. It would be useful to reword the relevant parts to avoid misunderstandings.  *** Original review: This paper proposes a method for few-shot classification of novel categories that belongs to the family of data augmentation. Specifically, the problem of few-shot classification requires constructing a classifier to distinguish between N new classes, of which only a few examples are available. The data augmentation approach to the problem aims to generate or synthesize additional ‘training’ examples for learning this N-way classifier, directly addressing the data insufficiency which is the cause of the difficulty of this setup. The proposed method, named Delta-encoder, can be used to generate additional examples of a new category given only a single ‘anchor’ example of that category. Intuitively, it works by creating a latent representation of a transformation that can be performed to the anchor example to transform it into something that still plausibly belongs to the same class as the anchor. In other words, it learns what transformations are ‘allowable’ for maintaining the ‘class-yness’ of the anchor example. This idea is the same as in [17], but the authors here propose a different method for achieving it. Notably, while [17] applied linear offsets to the anchor to generate new examples, this proposed method employs a non-linear function for learning these allowable transformations.  More specifically, Delta-encoder has an auto-encoder structure where the encoder takes a pair of examples that are known to belong to the same class and is encouraged by its training objective to use the latent space for encoding the ‘deformation’ between those two training examples. Specifically, this latent representation in addition to one of the initial two training examples are used by the decoder to reconstruct the other example. The dimensionality of the latent code is small enough to disallow this structure to function as a standard autoencoder where one of the two examples is merely reconstructed from itself, and the other is ignored. Therefore, what the latent code learns is how one of the two examples needs to be modified in order to be transformed into the other. The hope is that these ‘allowable class-preserving deformations’ that are learned between examples of training classes also are useful class-preserving deformations between instances of new classes.    Pros - Well-written, clearly conveys the problem setup and the proposed method - The proposed model is interesting and novel, to the best of my knowledge, and tackles an important problem - They show experimentally that this method performs very well on a wide range of standard datasets. - The section on ablation studies justifies the incremental changes performed on top of a Denoising Autoencoder model (for zero-shot learning) that was the inspiration for the Delta-Encoder.  Cons - While [17] and [43] seems to be the most related works, there is no experimental comparison to these methods. I feel this is quite important, and having this additional data point would significantly impact my opinion. - While a “linear offset” version is experimented with in the ablation studies section, I find that the particular linear offset version used there is a particular special case. Specifically, instead of computing the latent code as the vector difference between the anchor point and the input, this could be computed as the output of a linear layer that these two vectors as input (with a trainable weight and bias). I think it would be useful to quantify how well the best possible linear version fairs against Delta-Encoder. - While the paper overall is clearly written, I found lines 126-129 somewhat unclear. In particular, if I understand correctly these lines describe how the Delta-encoder is used at few-shot learning time. In particular, at this time, a single example of each new class is used as the anchor (if k > 1 examples are available of a class, they will in turn be used as the anchor) and the decoder produces additional examples that are hypothesized to belong to this new class, by taking as input this anchor and a Z (the latent code that encodes the deformation to be applied to the anchor). The part that is unclear to me, which is described by these lines, is how this Z is obtained. The quoted lines explain that a number of pairs of examples are encoded into Z’s, where each pair belongs to a training class (not one of the N new classes). This gives a set of { Z_i }’s. The part that is missing is how these are combined to form the Z that will condition the generation of examples of a new class. Is it an average that’s taken? In that case, would it make sense to just take the average over all encoded training pairs (which can be computed during the autoencoder training). - It should be noted that the representation-learning architecture used here is significantly larger than it is in many methods being compared to, casting doubts about the apples-to-apples-ness of these comparisons. Maybe an additional asterisk or note can be included in the table to indicate this (similar to how other aspects are indicated). In particular, Meta-Learner LSTM, MAML, Matching Networks, and Prototypical Networks all use a simple 4-layer CNN architecture.   Suggestions - I think that including a description of the overall algorithm in the section that explains the method would really aid the reader in understanding the process. For example something like the following: Given a dataset, the following phases are performed. Phase 1: Train a standard classifier on the set of training classes (often referred to as the meta-training set) of the dataset (all-way classification). These learned weights function as the pre-trained representation used in what follows. Phase 2: Train the encoder and decoder weights. As I understand it, this amounts to randomly sampling pairs of examples that have the same label, and training the autoencoder as described above. Phase 3: Few-shot learning phase. At this point, a ‘test episode’ is presented that contains k examples of each of N new classes. Each of these k examples of a given class will in turn be considered as the ‘anchor’ and the decoder will generate additional images of that class given this anchor and the latent code representing a deformation. Then, given all (real and synthesized) examples, an N-way classifier is trained for this episode. - Regarding Figure 2: A more descriptive and less confusing name for phase 2 (on the right of the figure) might be 'few-shot learning phase' instead of ‘sample synthesis phase’. In particular, as I understand it 'sample synthesis' is the process of decoding, which happens in both phases. I think the key part that distinguishes the two phases is that the former trains the encoder and decoder, while the latter applies the decoder to generate examples of new classes, given only an anchor for these classes.  - In related work, according to the proposed distinction between metric learning and meta-learning, Prototypical Networks should be in this meta-learning category too, instead of the metric learning one. Its training objective follows exactly the same structure as Matching Networks, the only difference being the choice of the conditional classifier's form.


Scaling the Poisson GLM to massive neural datasets through polynomial approximations
Recent advances in recording technologies have allowed neuroscientists to record simultaneous spiking activity from hundreds to thousands of neurons in multiple brain regions. Such large-scale recordings pose a major challenge to existing statistical methods for neural data analysis. Here we develop highly scalable approximate inference methods for Poisson generalized linear models (GLMs) that require only a single pass over the data. Our approach relies on a recently proposed method for obtaining approximate sufficient statistics for GLMs using polynomial approximations [7], which we adapt to the Poisson GLM setting. We focus on inference using quadratic approximations to nonlinear terms in the Poisson GLM log-likelihood with Gaussian priors, for which we derive closed-form solutions to the approximate maximum likelihood and MAP estimates, posterior distribution, and marginal likelihood. We introduce an adaptive procedure to select the polynomial approximation interval and show that the resulting method allows for efficient and accurate inference and regularization of high-dimensional parameters. We use the quadratic estimator to fit a fully-coupled Poisson GLM to spike train data recorded from 831 neurons across five regions of the mouse brain for a duration of 41 minutes, binned at 1 ms resolution. Across all neurons, this model is fit to over 2 billion spike count bins and identifies fine-timescale statistical dependencies between neurons within and across cortical and subcortical areas.

=== Update after author rebuttal === Thanks to the authors for their rebuttal. I have updated my score. One comment about the fully coupled GLM: since the validation data was taken from a specific portion of the experiment (minutes 45-46), then temporal non-stationarities in the data (e.g. changes in arousal state, adaptation, etc.) will make it harder to generalize. Instead, I would recommend that the authors hold out brief snippets of data throughout the experiment to use for validation and testing.  === Summary === This paper develops a method for scalable inference methods for Poisson generalized linear models (GLMs). These are particularly prevalent in the neuroscience literature. The authors use the polynomial approximate sufficient statistics (PASS-GLM) method to perform approximate Bayesian inference in these models. They apply this method to two neuroscience datasets: modeling primate retinal ganglion cell responses to white noise (here, the authors validate the approximate methods using exact inference) as well as a dataset of spontaneous activity recorded across brain regions in the mouse.  === Review ===  -- Clarity -- The paper is clear and well written. In particular, I appreciated the thorough background section, especially the comparison with the MELE.  -- Originality -- While the applications in neuroscience are new, much of the methods in this paper are directly taken from the PASS-GLM paper. At times, I found it hard to distinguish what (if any) of the method was new, versus deriving the PASS-GLM method for Poisson GLMs specifically.  Note that although the experiments in the PASS-GLM paper focus on logistic regression, that paper makes it clear that the method is appropriate for any model in the GLM family, including Poisson regression.  Therefore, it seems like much of the novelty of this paper is in the applications (specifically, the application to the large neuropixels dataset recorded across multiple mouse brain regions). Unfortunately, this occupies a small fraction of the overall paper.  -- Significance -- In this paper, the authors state (in the introduction) that their contributions are: (a) deriving the expressions for the MLE, MAP, and approximate posterior of a Poisson GLM using the PASS-GLM method, (b) validate the estimators on simulated data and retinal data, (c) introduce two novel extensions of the method, an analytic marginalization when using quadratic approximations and an adaptive procedure for selecting the interval of the polynomial approximation, and (d) apply these methods to study a large dataset recorded across multiple brain regions.  Given that most of the method is derived from the ideas in the PASS-GLM paper, I think (a) and (b) are less significant results.  Moreover, it is hard to judge the significance of the extensions to the method (c). For example, the authors did not report compare their results on using the adaptive intervals in section 4.2 to using fixed intervals, instead simply stating that they "find that this procedure provides large improvements" (line 185). Large improvements for which application? Simulated data? Retinal data? Mouse data? If the authors had a figure demonstrating this large improvement on real datasets with the adaptive interval, that would greatly increase the significance of this particular contribution.  Finally, while the application to mouse data (d) is certainly novel, it is equally hard to assess the significance of these findings. Namely, I think the authors should explicitly answer the following question: what scientific question would neuroscientists be able to answer with the approximate inference Poisson GLM that they could not answer otherwise? For example, am I unable to fit the GLM to the dataset using current techniques? Will it just take longer? How much longer? I would appreciate if there was a rough sense of how long it would take to do the exact GLM given N neurons and T time bins, compared to the approximate version.  This presupposed that I even want to fit a massive GLM. The application to the mouse dataset, to me, suggests that this might not be the best idea, as I was quite surprised at how much the GLM overfit to the dataset. The authors report that the model outperforms a homogenous GLM on only 65% of the neurons. Does this mean that the model is strongly overfitting? Shouldn't the hyperparameter search for the regularization parameters have dealt with this? Where was the 1 minute of held out data taken from (randomly throughout the recording, or at the beginning or end?). What if the authors did not fit coupling parameters between all neurons, instead fitting separate GLMs to each area? In that case, are there significant differences in predictive power? What if the full model was uncoupled (only spike history filters)? Also, I would have appreciated more interpretation of the model filters and coupling strengths presented in Fig 4d and 4c. Do these tell us something we did not already know? Do they look significantly different if I took subsets of neurons or data (such that I could do exact inference)?  I can appreciate if the authors' goal was to demonstrate that the approximate GLM works on large datasets, rather than analyze this particular GLM model fit, but it is hard to make sense of the significance of being able to fit a large GLM without more exploration of these details.  === General comments === This paper does a good job of taking a method recently published in the machine learning community (PASS-GLM) and shows how it may be relevant for neuroscience applications.  What I found most difficult about this work is that it is hard to tell which audience this is best suited for. I does not seem like the work is really about novel extensions to PASS-GLM that would be of interest to the machine learning community. Instead, it seems to target those in the neuroscience community who would be interested in building large scale GLMs of their data. However, for a neuroscientist, interested in applications, I think it is difficult to work through the derivations in sections 3 and 4 without a clear understanding of what the approximate GLM buys you, beyond being able to fit a GLM to a large recording (and it is unclear what that buys you in terms of new scientific understanding).  I think the paper could be improved by focusing more on applications to neural datasets, with more rigorous comparisons for the new ideas presented in this paper, as well as clear statements about what the approximate GLM buys you in terms of being able to ask new scientific questions.  The derivation of the approximate Poisson GLM (section 3), while valuable for those interested in applying the method, is harder to slog through when it is unclear what the method buys you (speaking as a neuroscientist). Perhaps focusing on applications and gains over previous methods first, and using that to motivate the derivation, would be more worthwhile for the community.


Do Less, Get More: Streaming Submodular Maximization with Subsampling
In this paper, we develop the first one-pass streaming algorithm for submodular maximization that does not evaluate the entire stream even once. By carefully subsampling each element of the data stream, our algorithm enjoys the tightest approximation guarantees in various settings while having the smallest memory footprint and requiring the lowest number of function evaluations. More specifically, for a monotone submodular function and a p-matchoid constraint, our randomized algorithm achieves a 4p approximation ratio (in expectation) with O(k) memory and O(km/p) queries per element (k is the size of the largest feasible solution and m is the number of matroids used to define the constraint). For the non-monotone case, our approximation ratio increases only slightly to 4p + 2 o(1). To the best or our knowledge, our algorithm is the first that combines the benefits of streaming and subsampling in a novel way in order to truly scale submodular maximization to massive machine learning problems. To showcase its practicality, we empirically evaluated the performance of our algorithm on a video summarization application and observed that it outperforms the state-of-the-art algorithm by up to fifty-fold while maintaining practically the same utility. We also evaluated the scalability of our algorithm on a large dataset of Uber pick up locations.

This is a clearly written and well argued paper that improves the known state of the art algorithms for the specific problem of streaming submodular optimization subject to multiple matroid (or matchoid) constraints. The authors improve the running time and either match or improve the approximation factors of known algorithms.  The experimental results can be significantly strengthened.   One downside of the work is that the theoretical improvement is relatively small, however in my mind this is outweighed by the fact that the algorithm is clean, seemingly universal (subsuming previous work), and practical.  On the technical side, the main contribution is Theorem 4, which gives a parametrized approximation guarantee for the Sample-Streaming algorithm. This is then combined with known results, and tuned settings for the parameters to obtain the stated results for monotone and non-monotone functions.  The proof is involved, it would have been good to provide a high level intuition picture in the Appendix (or if possible in the main body) of why this approach succeeds where previous methods have failed.   More notes: - Spell check -- l176 (independence), l248 (diverse),  - In the experimental section (line 265), it is worth noting explicitly that the proposed algorithm performs _worse_ than the SeqDPP baseline on OVP videos, and on half of the YouTube videos. A sentence is also warranted to describe the performance versus local search, which has a strictly worse approximation ratio, but performs comparably here.  - While it's nice to see absolute running time comparisons in Figure 1c, a much more meaningful metric would be the number of oracle calls, since this is precisely what the proposed method is trying to optimized. Moreover, such a metric would be independent of machine speed, or programming prowess, and can be used as a basis for comparison in the future. I strongly urge the authors to provide such a plot. Rebuttal Question: how does the number of evaluations scale between different methods?  - In my mind Section 4.2 should either be cut, or augmented with a stronger baseline. As mentioned above, simply reporting running time numbers without a baseline is not very meaningful, and future researchers would not be able to compare their methods against them.


Enhancing the Accuracy and Fairness of Human Decision Making
Societies often rely on human experts to take a wide variety of decisions affecting their members, from jail-or-release decisions taken by judges and stop-and-frisk decisions taken by police officers to accept-or-reject decisions taken by academics. In this context, each decision is taken by an expert who is typically chosen uniformly at random from a pool of experts. However, these decisions may be imperfect due to limited experience, implicit biases, or faulty probabilistic reasoning. Can we improve the accuracy and fairness of the overall decision making process by optimizing the assignment between experts and decisions? In this paper, we address the above problem from the perspective of sequential decision making and show that, for different fairness notions in the literature, it reduces to a sequence of (constrained) weighted bipartite matchings, which can be solved efficiently using algorithms with approximation guarantees. Moreover, these algorithms also benefit from posterior sampling to actively trade off exploitation-selecting expert assignments which lead to accurate and fair decisions-and exploration-selecting expert assignments to learn about the experts' preferences. We demonstrate the effectiveness of our algorithms on both synthetic and real-world data and show that they can significantly improve both the accuracy and fairness of the decisions taken by pools of experts.

## Updates after feedback/discussion ## The theoretical framework proposed in this paper is interesting and lends itself to numerous extensions that could be pursued in future work.  Overall I think the authors convincingly demonstrate that their approach can improve utility over the case of random assignment.  The evidence for fairness improvement might run into opposition as the assignment strategy may be viewed as discriminatory in practice.  This would be worth further discussion in the paper.  It would also be interesting to see more examples of what happens when there aren't all that many more decision makers than items.  A typical situation is one where there are more individuals than judges, so each judge needs to be used each round.  However, not all individuals are expected to be processed in each round.    I also think the authors can do more to provide motivation and intuition for their strategy.  Here's my interpretation of the overall approach.  The result for the unconstrained optimization setting can be interpreted as approximating the rule in equation (3) through a strategic assignment of judges to individuals. The rule in equation (3) applies the same threshold to everyone regardless of their group. This rule holds everyone, regardless of group, to the same optimal threshold. One can interpret the suboptimality of random assignment as arising from a (judge, individual) combination that differs from the optimal thresholding rules in equations (3). There are two reasons for why the random assignment rule may be suboptimal. (i) If judges are systematically biased against some group, the random assignment rule would effectively apply a higher standard to one group than another. We know that a single common threshold rule is optimal, so random assignment is suboptimal. (ii) Even if judges don't use group-specific thresholds, random assignment may fail to agree with thresholding at the optimal threshold c in equation (3). We can interpret the approach outlined in the paper as bringing us closer to the rule in equation (3) (equal standard) in the unconstrained case. We can interpret the results for the constrained case as bringing us closer (than random assignment) to the rule in equation (4), that optimally holds groups to different standards.  ## Original review ## Overview:  This paper considers the following problem. Suppose that there are m judges available to make decisions about n < m individuals (per round).  Individuals are described by (X, Z, Y), where X are features, Z is a group indicator (binary), and Y is an outcome.   Judges make decisions to maximize immediate utility by thresholding the common (and assumed known) conditional probability function p(x, z) = P(Y = 1 | X, Z) at judge-specific and group-specific thresholds.  What is the optimal way of assigning individuals to judges in each round so as to maximize utility subject to fairness (DI, TPR, TPR/FPR) constraints?  The authors show that the unconstrained problem can be solved efficiently as a bipartite matching between judges and individuals.  The constrained problem  can be reduced to bounded color matching. Theoretical results are provided showing that posterior sampling can achieve sublinear regret in the cases where the judge-specific thresholds are unknown and need to be estimated across rounds.  Empirical evaluations demonstrate performance of the proposed method in comparison to the more common strategy of random assignment of cases to judges.    Comments:   Overall I think this paper does a pretty good job from a theoretical standpoint of analyzing a toy problem that has some relevance to practice.   The authors readily admit that assuming all judges have access to the the same true probability function for decision making is highly idealized.  I wish the authors did more in either the discussion of the theory or the empirical results to more clearly characterize where the gains are coming from.  In terms of practical implications, the most relevant empirical comparison is between random assignment and the unknown thresholds case.  As I elaborate on below, I would like the authors to expand on this comparison to clarify their ideas.    - While DI as in (11) depends on there being 2 groups, it seems like for a more general definition of DI the results generalize to to non-binary Z.  Is this the case?  - The optimal threshold decision rules have the form given in (3) and (4) only if you assume that the probability function p(x,z) has positive density (see original Corbett-Davies paper).  If you have point masses in p(x,z), the optimal rule winds up being a randomized one.    - Discussion of COMPAS data: Individuals in bail proceedings are considered "defendants", not "offenders".  They have not yet been convicted of any crimes, and thus have not formally "offended".    - What is the precise definition of "Disparate impact" used in the experiments section?   - Your definition of “bias” in the sense of group-specific thresholds isn’t consistent with the discussion of fair classification/optimal threshold rules.  i.e., the optimal fair decision rule under those fairness constraints would have the judges applying group-specific thresholds.  So in what sense is setting unequal thresholds in your simulation “biased”?    - How does the performance degrade with the fraction of “biased” judges, and the degree of bias?  You compare in the COMPAS example for one “bias” level and 0 vs 50% “biased” judges, but I’m not sure if this is presenting a complete picture.  In what situations will the proposed strategy lead to improvement?    - When presenting the bar charts for the 50% biased case, how are infeasible solutions being treated?  Are you reporting the DI/utility omitting any infeasible solutions?  How big are the error bars on the utilities?  - By what mechanism is the DI being reduced in the experiments?  Is it because release rates are changing considerably across the Random, Optimal, Known and Unknown scenarios?  I’m especially interested in the answer to this question for the Random vs. Unknown scenario case.  If “release” rates are changing significantly, I don’t feel that the empirical evaluation is presenting an apples-to-apples comparisons.


Algorithms and Theory for Multiple-Source Adaptation
We present a number of novel contributions to the multiple-source adaptation problem. We derive new normalized solutions with strong theoretical guarantees for the cross-entropy loss and other similar losses. We also provide new guarantees that hold in the case where the conditional probabilities for the source domains are distinct. Moreover, we give new algorithms for determining the distributionweighted combination solution for the cross-entropy loss and other losses. We report the results of a series of experiments with real-world datasets. We find that our algorithm outperforms competing approaches by producing a single robust model that performs well on any target mixture distribution. Altogether, our theory, algorithms, and empirical results provide a full solution for the multiple-source adaptation problem with very practical benefits.

Summary of the paper:  This paper is interested in the problem of multiple-source adaptation. More precisely it is interested in the setting where a good model and an estimation of the distribution is available for several source domains. The goal is then to learn a good combination of the models to accurately classify target examples drawn from a mixture of the source domains. On the one hand the paper extends the theoretical results of [Mansour et al., 2008] to the stochastic setting where there exists a distribution over the joint input-output space. On the other hand it proposes an algorithm to correctly combine the models available from the different domains. The performance of the approach is empirically demonstrated on several tasks.  Main comments:  Pros:  - The paper is very well written and easy to read  - The setting considered is more general than existing works  - The proposed algorithm to estimate the target mixture is new, theoretically sound (Lemma 4) and works well in practice   Cons:  - The theoretical results are a bit incremental [Mansour et al. 2008]  - The practical usefulness of the method is not so clear as it requires an estimate of the distribution and a model for each domain  Detailed comments:  This paper is well written and, despite being quite technical, is easy to follow.  In the experiment displayed in Figure 1 and introduced Lines 246 to 251, I would have appreciated to see the results of the predictor used in Mansour et al. [2008]. Even it is not a real solution, it remains a good baseline to assess the limit of the method considered, that is it is probably one of the best achievable results using distribution-weighted combination of the source models.  The theoretical results are largely based on the previous work of Mansour et al. [2008] and mainly consist of an extension from the deterministic to the stochastic setting. In particular it considers a more realistic setting where the conditional probabilities of each output given an input might be different for the source and the target domain.  One of the main drawbacks of the proposed approach is that it assumes that the underlying distribution of the inputs is available for each domain which might not be very realistic in practice. This issue can be overcome by using a large number of unlabelled data from each domain to estimate the underlying distribution. In which kind of applications would this kind of setting (a good model and a set of unlabelled examples for each domain) be reasonable?  Some other minor comments:  - In the main paper, Line 111: I would have appreciated to have a bit more intuition about the meaning of \epsilon_T and not a simple definition, that is what does it represent or account for?  - In the supplementary, in the inequality following Line 440: It should be \hat{h} rather than h.  After the rebuttal:  The rebuttal solved my concerns regarding the practical usefulness of the method. However I still think that the contribution is a bit incremental and I agree with Reviewer 1 that a more extensive empirical evaluation would have been beneficial.


Statistical and Computational Trade-Offs in Kernel K-Means
We investigate the efficiency of k-means in terms of both statistical and computational requirements. More precisely, we study a Nyström approach to kernel k-means. We analyze the statistical properties of the proposed method and show that it achieves the same accuracy of exact kernel k-means with only a fraction of computations. Indeed, we prove under basic assumptions that sampling √ n Nyström landmarks allows to greatly reduce computational costs without incurring in any loss of accuracy. To the best of our knowledge this is the first result of this kind for unsupervised learning.

Summary  The paper investigates the kernel k-means problem, proposing a new approximation of the method based on Nystrom embeddings. In particular, both the statistical accuracy and computational efficiency of the proposed approximation is studied. The main contribution is the derivation of a theoretical bound limiting the error cost; the proposed method can achieve such theoretical guarantee with a computational cost of (O(\sqrt{n})). In practice, this implies that using \sqrt{n} points (with n being the original sample size) is enough to obtain a good enough approximation. The authors also propose an approach to choose the points while ensuring a balance trade off between preserving a good approximation and a small complexity (dictionary size).  Novelty and Clarity  The theoretical results are interesting and provide new insights into the topic. The writing style is easy to follow and the mathematical notation is also clear.   Overall opinion  The paper is technically correct and provides a new contribution to the field. The experimental section could be improved and some points should be clarified. The submission would be at a good level if the points below are clarified.   Major and minor comments are provided below   Major comments  - Proof of Lemma 1 and Lemma 2 could be provided in the supplementary (or at least a sketch)  - In the experiments it is not clear how the parameters, including the number of clusters, are selected.   - It would be beneficial to provide more detailed information on the runtime to further support the benefit of performing the approximation  - It would be beneficial to provide a more exhaustive comparison (and corresponding detailed information) with the competitors. This seems to be only slightly mentioned for the MNIST8M experiment  Minor comments  - The acronyms JL and RLS seem to be used without being introduced  - Section 4: The Lloyd’s algorithm reference is missing  - Algorithm 1 can be reformulated to be presented in a more structured form  - Figure 2: Legends and titles as well as a short caption could be added


Co-regularized Alignment for Unsupervised Domain Adaptation
Deep neural networks, trained with large amount of labeled data, can fail to generalize well when tested with examples from a target domain whose distribution differs from the training data distribution, referred as the source domain. It can be expensive or even infeasible to obtain required amount of labeled data in all possible domains. Unsupervised domain adaptation sets out to address this problem, aiming to learn a good predictive model for the target domain using labeled examples from the source domain but only unlabeled examples from the target domain. Domain alignment approaches this problem by matching the source and target feature distributions, and has been used as a key component in many state-of-the-art domain adaptation methods. However, matching the marginal feature distributions does not guarantee that the corresponding class conditional distributions will be aligned across the two domains. We propose co-regularized domain alignment for unsupervised domain adaptation, which constructs multiple diverse feature spaces and aligns source and target distributions in each of them individually, while encouraging that alignments agree with each other with regard to the class predictions on the unlabeled target examples. The proposed method is generic and can be used to improve any domain adaptation method which uses domain alignment. We instantiate it in the context of a recent state-of-the-art method and observe that it provides significant performance improvements on several domain adaptation benchmarks.

This paper presents a co regularization approach to domain adaptation. They build upon the "VADA" framework by A dirt-t approach to unsupervised domain adaptation. In particular, the VADA framework trains feature extractor 'g' and classifier 'h' by minimizing: distributional distance between features extracted from the source and target domains, label loss on source domain, and entropy + a smoothness operator over the output function (g*h) on the source and target domains.   The contribution of this paper is to first point out that alignment between source and target distributions is possible even with a 'g' that does not align the class distributions across source and target, and vice versa. To this end, the paper proposes to use co-regularization: learn two feature extractors g1 and g2 and two classifiers h1 and h2, via the VADA objective function, while also minimizing the distance between g1*h1 and g2*h2 and maximizing the distance between g1 and g2 for diversity. The authors claim that this can help rule out incorrect alignments.   Cons: 1. This paper argues in section 1 and figure 1 how alignment is not guaranteed to reduce target error and vice versa. However, with their co-regularization framework, they do not rigorously show (either theoretically or with controlled empirical results) that their approach indeed results in better alignment. The arguments on lines 112-116 and 123-128 are quite hand-wavy. This is indeed my biggest gripe with the paper because it is not quite clear how is the co-regularization actually helping in this case. In the co-regularization paper by Rosenberg and Bartlett, the use of two separate RKHS was shown to reduce Rademacher complexity, but what exactly is going on in this case?  2. More of a comment on above than a con: The experimental results show that even with \lambda_div=0, their framework provides improvements. The diversity in g1 and g2, in this case, is only coming from randomness (as the authors themselves point out on lines 242-245). In effect, this relies on the randomness of initialization and training along with non-convexity of the objective function to generate diversity. This needs to be explained and investigated further -- can we just rule out more and more "incorrect alignments" by randomly generating classifiers. If so, can we just make the feature generator better by learning 3 instead of 2 classifiers?  Question; 1. What set of parameters are finally used for testing on the target layer?   After author feedback: Upgraded my score to 6 provided the authors add their kNN explanation to the paper.


Learning Signed Determinantal Point Processes through the Principal Minor Assignment Problem
Symmetric determinantal point processes (DPP) are a class of probabilistic models that encode the random selection of items that have a repulsive behavior. They have attracted a lot of attention in machine learning, where returning diverse sets of items is sought for. Sampling and learning these symmetric DPP's is pretty well understood. In this work, we consider a new class of DPP's, which we call signed DPP's, where we break the symmetry and allow attractive behaviors. We set the ground for learning signed DPP's through a method of moments, by solving the so called principal assignment problem for a class of matrices K that satisfy K i,j = ±K j,i , i ≠ j, in polynomial time.

EDIT: I thank the authors for their good response. Although there is still work to make them ML workhorses, I am now convinced that signed DPPs are not unicorns. I encourage the authors to include this short paragraph about existence in the main text. Considering the other reviews and the whole response, I have now changed my overall rating.  # Summary of the paper and its contributions Finite determinantal point processes (DPPs) are statistical models able to capture both marginal relevance and joint interaction of samples. Most methodology papers have focused on using this interaction to model repulsiveness, which is usually ensured by the main parameter of the DPP being a positive semi-definite (p.s.d.) symmetric matrix. In this paper, the authors investigate DPPs with a mix of attractiveness and repulsiveness, using a special case of nonsymmetric p.s.d. matrices as kernel. The main contributions of the paper are theoretical, as the authors examine how to apply the method of moments to estimate the kernel matrix from data, and what the set of solutions look like. To do so, they exploit a connection to the algebra problem of finding a matrix with a given list of principal minors. The paper is a clear continuation of previous work on symmetric kernels, including references [7, 23].  # Summary of the review The paper is rather well-written (maybe a bit too technical by moments). Investigating nonsymmetric DPPs is very interesting and the authors deserve praise for tackling what seems to be a hard problem in a rigorous way. Moreover, the links with numerical algebra and graphs are interesting per se. However, my main concern is that the paper seems incomplete as such for an ML outlet: the solutions to the minor assignment problem are described, but under the strong assumption that a solution even exists. Relatedly, the existence of DPPs with so-called signed kernels is left undiscussed, and no example is given. I may have missed an important point and I am happy to revise my judgment if so, but currently this leaves the reader with the impression that we are potentially discussing the fine properties of statistical unicorns.  # Major comments - L143: there are two related problems that I think deserve much more space. The authors assume that a solution to the principal minor assignment problem exists. Relatedly, too little is said about the existence of a signed DPP with a given kernel matrix. There is a short mention L144 of reference [20], but I haven't been able to pinpoint a relevant result in [20], which is hard to read for an ML audience and seems to treat symmetric matrices anyway. Could you 1) give a sufficient condition for existence of signed DPPs? and 2) point me (and maybe explain in a few layman's words) to the relevant result in [20]? - L264-266: I am not convinced by these three lines that condition (3) is not a big assumption. In particular, I don't understand why it is relevant that the condition is satisfied with probability 1 for a nondegenerate random kernel matrix. If I want to apply a method of moments, I want to estimate a fixed but unknown matrix, arguments of the type "with high probability" do not enter the picture. Moreover, kernel matrices used in ML are very often structured and low-rank, for which I doubt that the mentioned random models would yield good representers. - p8, Algorithm 1: Gaussian elimination in GF_2 should be explained to an ML audience, and probably an example should be given. Providing code for the overall algorithm would further help. - Theorem 4: what happens is the assignment problem does not have a solution? How can an ML user detect that the algorithm is producing an inconsistent result? - Overall, an experimental section would be nice. Since existence is assumed, the properties of the method of moments should at least be numerically investigated when there is no underlying model. This is an extreme case of misspecification.  # Minor comments & typos - L53: some of the references referred to as maximum likelihood are not MLE methods. - L253: Condition (4) should probably read "Condition (3)". - p8, Algorithm 1: there are many misaligned indices when writing submatrices. - same: elimination is misspelled.


Decentralize and Randomize: Faster Algorithm for Wasserstein Barycenters
We study the decentralized distributed computation of discrete approximations for the regularized Wasserstein barycenter of a finite set of continuous probability measures distributedly stored over a network. We assume there is a network of agents/machines/computers, and each agent holds a private continuous probability measure and seeks to compute the barycenter of all the measures in the network by getting samples from its local measure and exchanging information with its neighbors. Motivated by this problem, we develop, and analyze, a novel accelerated primal-dual stochastic gradient method for general stochastic convex optimization problems with linear equality constraints. Then, we apply this method to the decentralized distributed optimization setting to obtain a new algorithm for the distributed semi-discrete regularized Wasserstein barycenter problem. Moreover, we show explicit non-asymptotic complexity for the proposed algorithm. Finally, we show the effectiveness of our method on the distributed computation of the regularized Wasserstein barycenter of univariate Gaussian and von Mises distributions, as well as some applications to image aggregation.

This paper presents a distributed algorithm for computing Wasserstein barycenters.  The basic setup is that each agent in the decentralized system has access to one probability distribution; similar to “gossip” based optimization techniques in the classical case (e.g. popularized via ADMM-style techniques) here the distributed algorithm exchanges samples from local measures along edges of a graph.  It seems this paper missed the closest related work, “Stochastic Wasserstein Barycenters” (Claici et al., ArXiv/ICML), which proposes a nonconvex semidiscrete barycenter optimization algorithm.  Certainly any final version of this paper needs to compare to that work carefully.  It may also be worth noting that the Wasserstein propagation algorithm in “Convolutional Wasserstein Distances: Efficient Optimal Transportation on Geometric Domains” (2015) could be implemented easily on a network in a similar fashion to what is proposed in this paper; see their Algorithm 4.  Like lots of previous work in OT, this technique uses entropic regularization to make transport tractable; they solve the smoothed dual.  The basic model for translating this problem to a network problem is similar to that in many other distributed optimization algorithm; you duplicate the unknown barycenter variable per vertex and add equality conditions along the edges.  In general, I found the idea in this paper to be compelling, and I appreciate the careful theoretical and experimental analysis.  I’m not sure the paper actually makes a strong case for network-based optimization (I’m guessing the experiments here were largely on a single machine than some huge network of computers), and I’d love to see truly large-scale experiments that better justify ML applications----but this can be a great topic for future work.   Other:  L40:  No need to capitalize “entropic”  L46, “privacy constraints:”  Is it clear the technique in this paper has any privacy-preserving properties?   L150:  [this comment is likely a mistake on my part but I’ll ask it anyway!]  I’m not sure I follow why sqrt(W)*p=0 implies that all the p’s are equal.  For example, consider the line graph with vertex labels 0-1-2-3-4-5-6.  Notice the label of every vertex on the graph is the average of the labels of its neighbors, but the signal is not a constant.  Why does that not happen for this paper?  To fix the issue on the previous line, I would think a simpler way to write the constraint is Dp=0, where D is the E x V incidence matrix that takes  one value per vertex and subtracts along edges; one way to obtain the graph Laplacian is by writing D^T * D = L.  I believe the rest of the algorithm would work out identically.  It seems a large part of sec 3.1-3.2 is a standard application of proximal methods (problem (P) is a standard one of course); perhaps it could be shortened if the authors need to expand their discussion elsewhere.  In particular, two different algorithms are given which are both at a level of generality greater than what is needed later in the paper.  Currently this section and sec 4 are quite dense and hard to follow relative to the other discussion.  Given that the optimization problem is entropically regularized, why in sec 3.2 is it chosen to use quadratic terms for the proximal algorithm?  I would have guessed entropy/KL would be a better fit here.  Figure 1 is a great idea!  It answer’s my main question, which is dependence of convergence on topology of the graph.  What caught me by surprise is that convergence for the star graph is so slow --- I would have expected faster “mixing time” (I know, not in the MCMC sense!) since all the distributions are two links away in the topology.  Is there intuition for why this doesn’t happen?  [took a look at the rebuttal and it's all reasonable to me --- this is a nice paper!]


Revisiting Multi-Task Learning with ROCK: a Deep Residual Auxiliary Block for Visual Detection
Multi-Task Learning (MTL) is appealing for deep learning regularization. In this paper, we tackle a specific MTL context denoted as primary MTL, where the ultimate goal is to improve the performance of a given primary task by leveraging several other auxiliary tasks. Our main methodological contribution is to introduce ROCK, a new generic multi-modal fusion block for deep learning tailored to the primary MTL context. ROCK architecture is based on a residual connection, which makes forward prediction explicitly impacted by the intermediate auxiliary representations. The auxiliary predictor's architecture is also specifically designed to our primary MTL context, by incorporating intensive pooling operators for maximizing complementarity of intermediate representations. Extensive experiments on NYUv2 dataset (object detection with scene classification, depth prediction, and surface normal estimation as auxiliary tasks) validate the relevance of the approach and its superiority to flat MTL approaches. Our method outperforms state-of-the-art object detection models on NYUv2 dataset by a large margin, and is also able to handle large-scale heterogeneous inputs (real and synthetic images) with missing annotation modalities.

Main Idea: The work targets a special case of multi-task learning (MTL) referred to as primary MTL where the goal is to use auxiliary tasks to improve performance on a primary task. The primary task considered is object detection with scene classification, depth prediction, surface normal estimation being the auxiliary tasks.  The authors introduce a Residual Auxiliary Block (ROCK) which can be used to augment existing deep network-based object detection pipelines. ROCK is proposed as an alternative to the standard MTL approach used in object detection literature where a backbone feature extractor is followed by multiple task-specific heads e.g. classification, bounding box regression, and segmentation heads in MaskRCNN. The paper comprises of justification of various design choices in ROCK and ablation studies to quantify the benefit of those choices.   Strengths: - The authors do a good job of zooming in on a specific problem (primary MTL) that they are trying to solve. Writing and experiments are appropriately designed around that problem.  - Fig 2 summarizes the model well. - The writing is clear and Sec. 2 and 3 helps the reader understand the authors' motivation behind various design choices that went into ROCK.  - Ablations in Tables 1 and 3 demonstrate the benefits of using auxiliary annotations, encoders, and feature fusion in ROCK over the standard MTL baseline on the NYUv2 dataset.  - Table 4 tries to answer an interesting question - Is it better to have more images with one kind of annotations or fewer images with annotations for a wide variety of tasks? The numbers indicate that 70% of the data with multiple task annotations is sufficient to achieve performance comparable to a detector trained on 100% data without MTL.   Weaknesses: Writing - Primary MTL should be differentiated from alternative goals of MTL (such as improving the performance of all tasks or saving memory and computational cost by sharing computation in a single network) early on in the abstract and introduction - In the abstract, a point is made about residual connections in ROCK allowing auxiliary features to explicitly impact detection prediction. This needs to be contrasted with standard MTL where the impact is implicit.  - The claim about ROCK handling missing annotations modalities is unclear. The MLT dataset needs to be described in Sec 4.3.  - The introduction describes Transfer Learning (TL) and Fine-tuning (FT) as sequential MTL. I do not completely agree with this characterization. TL is a broader term for the phenomenon of learnings from one task benefitting another task. Fine-tuning is a sequential way of doing it. Standard MTL is a parallel means to the same end.  - The first 2 paragraphs of introduction read like literature review instead of directly motivating ROCK and the problem that it solves.  - Need to describe Flat MTL and include a diagram similar to Fig 2 that can be directly visually compared to ROCK.  - Fig 1 is not consistent with Fig 2. Fig 2 shows one encoder-decoder per auxiliary task whereas Fig 1 shows a single shared encoder-decoder for multiple tasks.  - L88-89 try to make the case that ROCK has similar complexity as the original model. I would be very surprised if this true because ROCK adds 8 conv layers, 3 pooling layers, and 1 fusion layer. Inference timings must be provided to make this claim.   Weaknesses: Experiments - Ablation showing the contribution of each auxiliary task on object detection performance in ROCK as well as in standard MTL. This can be done by comparing the full model (DNS) with models where one task is dropped out (DS, NS, DN).  - Cite MLT dataset in Table 2 caption and describe Geo, presumably some kind of geometric features.  - More details are needed for reproducibility like backbone architecture, activations between conv layers, number of channels, etc.   Weaknesses: Originality - While primary MTL setting considered in this work is very useful, especially in data-deficient domains, the main contribution of this work, the ROCK architecture, comes across as a little incremental.  Summary My current assessment is that in spite of the weaknesses, the model is simple, reasonably well motivated and shows decent performance gains over appropriate baselines. Hence I am currently leaning towards an accept with a rating of "6". I am not providing a higher rating as of now because the writing can be improved in several places, an ablation is missing, and the novelty is limited. My confidence score if 4 because I have only briefly looked at [19] and [35] which seem to be the most relevant literature.    --Final Review After Rebuttal--  The authors did present an ablation showing the effect of different tasks on multitask training as requested. A significant number of my comments were writing suggestions which I believe would improve the quality of the paper further. The authors have agreed to make appropriate modifications. In spite of some concern about novelty, I think this work is a valuable contribution towards an understanding of multitask learning. All reviewers seem to be more or less in agreement. Hence, I vote for an accept and have increased my rating to 7 (previously 6).


Computationally and statistically efficient learning of causal Bayes nets using path queries
Causal discovery from empirical data is a fundamental problem in many scientific domains. Observational data allows for identifiability only up to Markov equivalence class. In this paper we first propose a polynomial time algorithm for learning the exact correctly-oriented structure of the transitive reduction of any causal Bayesian network with high probability, by using interventional path queries. Each path query takes as input an origin node and a target node, and answers whether there is a directed path from the origin to the target. This is done by intervening on the origin node and observing samples from the target node. We theoretically show the logarithmic sample complexity for the size of interventional data per path query, for continuous and discrete networks. We then show how to learn the transitive edges using also logarithmic sample complexity (albeit in time exponential in the maximum number of parents for discrete networks), which allows us to learn the full network. We further extend our work by reducing the number of interventional path queries for learning rooted trees. We also provide an analysis of imperfect interventions.

Authors use single vertex interventions and declare the variables whose distribution changes with the intervention to be the descendants. Based on this, they can recover the transitive reduction of a causal graph with n interventions. The advantage of this approach, although it increases the number of interventions from log^2(n) to n compared to [13], is that it requires much less number of samples, since intervention sets are all of size 1.   The fact that an intervention changes the distribution of the descendants even in the presence of latents has been known and is lately used in [13]. Accordingly, the overall approach is not surprising, when the goal is to minimize the number of interventional samples. This approach is presented using what the authors call "a path query". What the authors call a path query seems equivalent to a conditional independence test on the post-interventional distribution when we use a stochastic intervention. Algorithm 3, the algorithm for recovering the true causal edges, gradually increments the parent set of each node by including the newfound parent to the current parent set. This algorithm however requires exponential number of samples, and its advantage is not clear to me over the algorithms of [13].   Given this high level idea, authors analyze the number of samples required to reliably run these tests. I am happy to see this analysis and this is definitely an important direction for the causality literature. However the presented analysis have some issues. Most importantly, the authors use DKW inequality, which is useful for continous variables, in the proof of Theorem 1. However, since they are using empirical counts to approximate a discrete distribution, they could have used Chernoff bounds. In fact, had they used Chernoff bounds, I believe they would get a 1/sqrt(gamma) dependence, whereas with the current analysis, they have a 1/gamma^2 dependence. (details below)   Another important issue with the submission is the title: The title is very misleading. The authors only find the transitive reduction of a causal graph. Not learn the whole Bayesian network. I recommend mentioning causal graph and transitive reduction in the title. It is also surprising to see that the authors did not choose causality as primary or secondary topic for the submission.  Despite the not surprising algorithm and some issues with the analysis, I am leaning more towards acceptance. However, I would appreciate if the authors could respond to the questions I asked in the following and clarify certain points for me to make a better assessment.  Following are more details and my feedback/other questions for the authors:  What authors call a path query is simply checking if a descendants distribution changes or not. I understand that they are following up on [32], but calling this known test a "path query" makes the work seem disconnected from the existing causality literature. I would remove this language from the paper since it is simply checking if the underlying distribution of the other nodes changes with different interventions. Also, the test seems equivalent to a CI test, when we use a stochastic intervention on the support of the intervened node. Could you please comment on this and maybe add a discussion about this equivalence in the paper?  There are certain issues with the way background is explained. The authors state that Bayesian networks are also used to describe causal relations. hen they are called causal Bayesian networks. This is not correct. A Bayesian network is a graph according to which the joint distribution factorizes. Bayesian networks do not have causal interpretations. Please change the narrative accordingly in line 24 and also in line 100, where it is told that a Bayesian network can also be viewedd as a causal model or a causal Bayesian network. This narrative is misleading.  The definition of a v-structure in footnote 1 in page 1 should mention the fact that X, Z should not be adjacent.  The authors use "number of interventions" for the number of total experiments performed in order to realize a collection of interventions: If they perform a single intervention on a 5 state variable, they count this as 5 interventions. This arises slight confusion, especially when they are citing the performance of the related work. I would use "number of interventional samples" for this to avoid confusion. But this is not a major point.  In line 108, the probability expression should contain Xi as well, since authors use indicator Xi=xi on the right hand side.  The path query definition given in lines 120-122 is word by word taken from [32], which the authors cite elsewhere in the paper. Please add this citation here in the definition to signify where the definition is taken from.  In line 149, authors mention that their method is robust to latent confounders. This is correct since they only use interventional data and search for the desendants by looking at the invariances in the interventional distribution (This is in fact how I would describe this work and the algorithm rather than using the path query language. This would also make the paper more consistent with the existing causality literature). However, it is too important to simply mention in a single line. Please elaborate and explain why latent confounders do not affect any of the analysis in the paper (no need to respond here).  Proof of proposition 2 is not rigorous. Authors should use faithfulness. Please see the appendix of [13] for a similar proof. (I suspect interventional faithfulness to be necessary here). Please comment on this.  In Theorem 1, the authors simply estimate each interventional distribution from the empirical counts, and check if they are different enough. The analysis of this algorithm uses what is known as DKW inequality. This inequality is useful for continuous random variables. I don't understand why the authors have not used a Chernoff boumd. I believe with Chernoff bound, they should be able to get a 1/sqrt(gamma) bound, whereas with this inequality, they can only show 1/gamma^2 bound. The only change in the analysis would be that, since the difference in distributions is assumed to be gamma, you need a concentration of gamma/2 in each distribution. But this would only change the constant. Please check the analysis to verify this suggestion.  In Theorem 2, the authors assume that Xj are subGaussian. But how do we know P(Xj|do(Xi)) are also subGaussian? Please explain. Because the proof of theorem 2 uses this. The authors should state this as an assumption of Theorem 2. This seems necessary for this result.  Algorithm 3 operates by, for each node adding the parental nodes not already included in the transitive reduction. The algorithm seems correct. However, with the authors' definition of the number of interventions, this requires exponential in n number of interventions since they have to intervene on all the parents of every node. They acknowledge this elsewhere in the paper, but not including this fact in the theorem statement is a slighly misleading. It is not clear what the advantage of Algorithm 3 is, compared to the algorithms given in [13]. Please elaborate.   There are various grammatical errors and typos throughout. I recommend proofreading the submission. Following are what I could catch:  Line 5: any causal Bayesian networks -> any causal Bayesian network Line 8: by intervening the origin node -> by intervening on the origin node Line 23: are a powerful representation -> are powerful representations Line 33: the effect on Y by changing X -> the effect of changing X on Y Line 34: DAGs -> causal graphs Line 40: or assume that have one -> or assume that they have one Algorithm 2, Line 1: Intervene Xi -> Intervene on Xi


Distilled Wasserstein Learning for Word Embedding and Topic Modeling
We propose a novel Wasserstein method with a distillation mechanism, yielding joint learning of word embeddings and topics. The proposed method is based on the fact that the Euclidean distance between word embeddings may be employed as the underlying distance in the Wasserstein topic model. The word distributions of topics, their optimal transports to the word distributions of documents, and the embeddings of words are learned in a unified framework. When learning the topic model, we leverage a distilled underlying distance matrix to update the topic distributions and smoothly calculate the corresponding optimal transports. Such a strategy provides the updating of word embeddings with robust guidance, improving the algorithmic convergence. As an application, we focus on patient admission records, in which the proposed method embeds the codes of diseases and procedures and learns the topics of admissions, obtaining superior performance on clinically-meaningful disease network construction, mortality prediction as a function of admission codes, and procedure recommendation.

Summary ======= The authors present a Distilled Wasserstein Learning (DWL) method for simultaneously learning a topic model alongside a word embedding using a model/approach based on the Wasserstein distance applied to elements of finite simplices. This is claimed as the first such method to simultaneously fit topics alongside embeddings. In particular, their embeddings only exploit on document co-occurence rather than nearby co-occurence within a sequence (i.e. using word order information) such as with word2vec. The authors demonstrate the superiority of their embeddings against a variety of benchmarks on three tasks: mortality prediction, admissions-type prediction, and procedure recommendation, using a single corpus of  patient admission records where words are the international classification of diseases (ICD) ids of procedures and diseases.  There are a number of apparently novel features to their approach which they outline in their paper, namely:  * It is a topic model where observed word frequencies within a document are approximated as the *barycentres* (centre of mass) of a weighted sum over a low rank basis of topics (where these barycentres are with respect to some Wasserstein distance). This is a compelling idea, as the Wasserstein distance is a very natural one for probability distributions. * Development of an objective function, allowing alternating optimisation. One phase learning the documents as distributions of topics (the weights \Lambda) and the topics as distributions of words (the basis B). The other phase refining the word embeddings. * A distilling approach to avoid vanishing gradients when learning the weights \Lambda and basis B. This looks like a regularised version of the objective, which takes advantage of Sinkhorn distance and a fractional powers of the underlying embedding distance. * A smoothness regulariser on the parameters, \theta, of the embedding function, meaning it is not far from the previous iteration's value. * It is unclear whether this method will scale.   The paper is well written, and the method appears reproducible. Further, it is both dense in mathematical justification, and clearly references related work and foundational results. To my knowledge it is novel and represents a significant contribution which would be of interest to the field. I have a few concerns about the following aspects of the paper:  * There is no independent verification/clear justification that word order in the patient record data is less consistent than in natural language corpora. We rely on the superiority of this model over those that utilise word order information for their embeddings. * Sometimes the intuitive explanations fall a little short. * There is a heavy reliance throughout on foundational results from an arXiv paper [36] (hence not peer reviewed). * For the regularisation/distilling approaches, there don't appear to be any theoretical results assessing their influence on the convergence/optimality of the overall model. Moreover, they involve choosing hyperparameters whose influence it is difficult to assess other than empirically, * The evaluation is done on just one data-set (with three related tasks), and while the results show that the proposed method consistently outperforms other existing approaches the improvements are modest and there are no errorbars/significance results.    Nonetheless, I am recommending the paper for acceptance, as I feel the positive aspects outweight these negatives.   Detail ======  ### p1, lines 31-33 > ICD codes for a given patient admission is less relevant, as there is often a diversity in the observed codes for a given admission, and code order may hold less meaning  This could be more convincing. There is diversity in the words that appear around other words in natural language. Could you verify this on data?  ### p2, figure 1 I think it is possibly trying to do a little too much in one figure, or it might be the choice of perspective. Having read the paper from beginning to end, it is clearer, but it would have helped having the associated intuition earlier.  ### p3, lines 92-93 > Π(u, v) is the set of all probability measures on Ω × Ω.  I think there should be an addional requirement that the left and right marginals are u and v respectively, e.g. given some $$\gamma \in \Pi(u,v)$$ then $$\gamma(\Omega,A) = v(A) \quad \text{for all } A \subseteq \Omega$$  ### p3-p4, use of the word transfer > ...the transfer between different topics (and different documents) is achieved by the optimal transport built on the embedding-based underlying distance. For arbitrary two word embeddings, the more similar they are, the smaller ground distance we have, and more easily we can achieve transfer between them.  This is a little ambiguous and could be made clearer.   ### p7, table 1 Although the performance is consistently better the results only represent a seemingly modest improvement (especially considering the increased computational complexity).  ### p7, line 263 The variables P_m and R_m are not defined (although at a guess they are the summands of the precision and recall respectively.  ### p8, line 270, clarity > dispersed mainly accordingly to their connections with matched diseases.  Not sure what is meant by mainly (or how that is determined).  How is this to be evaluated. It is unclear how anyone other than a medical expert would be able to evaluate how meaningful this clustering is, without some additional information (even given the additional information in the supplementary materials).


Causal Discovery from Discrete Data using Hidden Compact Representation
Causal discovery from a set of observations is one of the fundamental problems across several disciplines. For continuous variables, recently a number of causal discovery methods have demonstrated their effectiveness in distinguishing the cause from effect by exploring certain properties of the conditional distribution, but causal discovery on categorical data still remains to be a challenging problem, because it is generally not easy to find a compact description of the causal mechanism for the true causal direction. In this paper we make an attempt to find a way to solve this problem by assuming a two-stage causal process: the first stage maps the cause to a hidden variable of a lower cardinality, and the second stage generates the effect from the hidden representation. In this way, the causal mechanism admits a simple yet compact representation. We show that under this model, the causal direction is identifiable under some weak conditions on the true causal mechanism. We also provide an effective solution to recover the above hidden compact representation within the likelihood framework. Empirical studies verify the effectiveness of the proposed approach on both synthetic and real-world data.

UPDATE after authors' rebuttal: Thanks a lot for your detailed answers, quite convinced by most of your arguments, you played the game well within the one page constraint! I increase my score from 6 to 7, hoping it would be enough to see your paper being presented at NIPS! If not (competition is strong here!), I strongly encourage you to still fight to publish it, clearly a key topic (I'm biased :)).  This work presents a method to decipher a causal link between 2 variables. The method relies on a re-coding of one of the variables (the cause) to obtain an 'optimal' link with the effect variable. The paper reads well, is well structured and easy to understand. I am puzzled as to whether it is a mere simple implementation of existing work or whether it is a truly novel (and then would well be worth presenting) method. The numerical results are not very convincing.  Comments:  - abstract: as a first pass, I was intrigued by the use of a hidden variable mentioned here, wondered whether it was unobserved. OK once reading the paper.  - introduction: l36 "right" -> define what the right order is. Is the variable necessarily ordered? Cannot be purely qualitative? This paragraph is quite interesting. Do you have a ref? If this is original work, may need further explanation. l45 (typo): ...maps THE cause to...of lower cardinality  - Before you introduce your work from l43, is the motivation strong enough? I am not entirely convinced at this stage...Moreover, it reads that a (simple) re-coding of the (true) cause is enough to distinguish the true causal relationship. It is not very intuitive. Careful, the Table of Figure 2 is not causal, only probabilistic (see text l60)  - Section 2, l72: f needs to be a surjection, right? You could enumerate them (how many are there for each unknow |Y^{\prime}| ? l79: not sure your example is a misdiagnosis example. l83: LOG-likelihood. l96: ...$ P(X) $, THE deterministic...  - l103: is the alternate maximisation equivalent to a joint maximisation or an approximation? Quality of the approximation then? l117: ...with $ y_0 $, where...Could you also explain the choice for y0?  - Bottom of p4: so in the end, (i) re-coding either X or Y and finding a 'good' probabilistic relationship with the target variable is enough? (ii) Moreover, it quantifies the 'correctness' of the model? (iii) What if neither X->Y, nor Y->X are correct?  - Section 3: does Th 1 generalise  the additive noise model assumption? THIS IS INTERESTING. So is Th 4.  - Section 4: l174, remove (HCR in short), l175 ...on both synthetic and real world data. Rephrase the next sentence.  - Your choice is synthetic data looks 'easy' as it is exactly your model. May need more general framework! You could then do some sensitivity of your method to breaking the assumption of Th 1?!  - l188: notation issue, do you mean {3,...,15} instead? Fig2: why don't you show performance below m=500? Strange performance for SA, ANM and DC. Worse than random guess??!! Define the decision rate.  - real world data: I am sure these data sets were used in some public challenges (see the work by I. Guyon), I doubt they get results as poor as the ones you show. How do you explain some method perform more poorly when m increases?  - References: consistency, with or without first names? (e.g. Jonas Peters, but B. Sch\"{o}lkopf)


Revisiting ( , γ, τ )-similarity learning for domain adaptation
Similarity learning is an active research area in machine learning that tackles the problem of finding a similarity function tailored to an observable data sample in order to achieve efficient classification. This learning scenario has been generally formalized by the means of a ( , γ, τ )−good similarity learning framework in the context of supervised classification and has been shown to have strong theoretical guarantees. In this paper, we propose to extend the theoretical analysis of similarity learning to the domain adaptation setting, a particular situation occurring when the similarity is learned and then deployed on samples following different probability distributions. We give a new definition of an ( , γ)−good similarity for domain adaptation and prove several results quantifying the performance of a similarity function on a target domain after it has been trained on a source domain. We particularly show that if the source distribution dominates the target one, then principally new domain adaptation learning bounds can be proved.

This paper is a theoretical look at domain adaptation / transfer learning problems through the lenses of similarity learning. The authors have extended an already established similarity learning theoretical framework to cases where the training and testing distributions differ. The authors rigorously prove the following in this paper:  - A (\epsilon,\gamma)-good similarity for a problem in a source domain is also is an (\epsilon + \epsilon', \gamma)-good similarity in a target domain, assuming the same landmark distribution on both the source and the target.  - They further extend this result in Theorem 2 when the landmark distributions are different in the source and target domains. In this case, the problem in the target domain becomes (\epsilon + \epsilon' + \epsilon'', \gamma) good.   Both \epsilon' and \epsilon'' are formally derived in the paper. The former is a product of the worst margin achieved on some instances and a distance between the source and target distributions. The latter is completely defined by the landmark distributions in the source and target.   The authors also provide bounds for the worst margin including another theoretical results (theorem 3) providing a tight upper bound for the true worst margin when estimated empirically.  I have found this work very interesting, quite rigorous but difficult to read. I felt that the authors were somehow under pressure to compress the content to fit under the 8 page limit. For instance, in Section 3.2.2, I recommend adding a bit more explanation in lines 162 to 165, explaining how the first difference in the equation in line 161 (or right below it; I think that numbering this equation might be a good idea) balls down to applying lemma 1 and imposes \epsilon' in theorem 2. \epsilon'' is derived from the last two terms in this equation on line 161. While things are all clear with the supplement, I think that putting pointers from the submission to the supplement will help the reader significantly. On line 170, I would refer back to \epsilon'' when discussing why it is a good thing to use the same landmark distributions in the source and target domains. In general, adding more intuition behind the precise mathematical construct will certainly help this paper.   The experimental section does illustrate the benefit of model adaptation. However, it would be good to have at least a second use case a bit more complex from the 2D Gaussian one that is proposed with different rotation angles as a way to control the similarities between the source and the target.       Beside these comments/suggestions, let me reiterate that I have found the work quite interesting and sound.


RetGK: Graph Kernels based on Return Probabilities of Random Walks
Graph-structured data arise in wide applications, such as computer vision, bioinformatics, and social networks. Quantifying similarities among graphs is a fundamental problem. In this paper, we develop a framework for computing graph kernels, based on return probabilities of random walks. The advantages of our proposed kernels are that they can effectively exploit various node attributes, while being scalable to large datasets. We conduct extensive graph classification experiments to evaluate our graph kernels. The experimental results show that our graph kernels significantly outperform existing state-of-the-art approaches in both accuracy and computational efficiency.

The paper proposes a kernel for graphs able to deal with discrete and continuous labels. In particular, the topology information of a graph is encoded at node level by a return random walk probability vector (each dimension being associated to a different walk length). This probability vector is obtained by classical equations used by random walk kernels for graphs (T. Gartner et al. classical LTKM paper) but just focussing on return random walks. Thanks to that the computational complexity can be reduced since only the entries on the diagonal of the powers of the transition probability matrix need to be computed. This can be done via eigen-decomposition of a rescaled version of the adjacency matrix. Notwithstanding that, for large  graphs  the computational complexity is still too high and Monte Carlo methods are suggested as a practical way to get estimates of the quantities of interest. Having obtained the (approximated) return probability vector for each node of the graph, an embedding of a graph combining also the original information attached to each node (via tensor product of kernels) is obtained by kernel mean embedding (proposed in a JMLR 2012 paper).  Again, the embedding is computationally too expensive for large graphs, so the paper proposes to exploit previous work on approximate explicit feature maps (NIPS 2008 paper by Rahimi & Recht) to get a moderate size explicit tensor representation of graphs.  All the proposed kernels exploit either a polynomial or a RBF transformation (exploiting random Fourier feature map for the approximated version). Experimental assessment is performed on classical benchmarking datasets involving just structural information or additional discrete and continuous information at node level.  Quality: The mathematical treatment is solid as well as all the concepts that are used as building blocks of the proposed approach. I just have few concerns on the following points:  i) one of the problems of random walks is tottering (Mahe et al., ICML 2004), i.e. the fact that short paths in the graph can be visited many times forward and backward by a walk, thus over representing that topological feature with respect to walks supported by longer paths. This problem is particularly severe for return walks. In the paper there is no discussion about this well known problem as well as there is no analysis of the impact of this problem on the quality of the obtained vectorial representations. ii) the properties of the return probability features (RPF) may not be preserved by their approximated versions. For example, isomorphism-invariance is not valid anymore since the same graph with different (or even same) representations can get different RPFs because of the Monte Carlo approximation. The paper does not mention this issues as well as does not present empirical evidence of the quality of the obtained representations. iii) since nonlinear transformations (polynomial and RBF) are used on top of RPFs and additional node information, it is not clear to me how fair is the experimental comparison. In fact, I guess that a fair comparison would imply the use of the very same kernels on top of the other kernels taken from the literature. It is not clear to me how much of the observed improvements  (in some cases marginal with respect to the state-of-the-art) are due to that. iv) concerning the experimental comparison versus graph kernels for continuous attributes, the author missed a recent proposal that is relevant and uses some of the paper datasets:  Giovanni Da San Martino, Nicolò Navarin, Alessandro Sperduti: Tree-Based Kernel for Graphs With Continuous Attributes. IEEE Trans. Neural Netw. Learning Syst. 29(7): 3270-3276 (2018).  Clarity: the paper is well written and can be followed with no problem by an expert in the field. Organisation of the material is good. The level of presentation allows the reader to both have a general idea of the flow of reasoning and some technical details that characterise the proposed approach.  Originality: the paper builds on results from the literature. The idea to just use the return probabilities of random walks is new to my knowledge, although it can be considered a bit derivative. Some effort has been put into the derivation of the theoretical results concerning RPFs and new proposed kernels. Overall I see some original contribution here.  Significance: I guess the main contribution of the proposed kernels is the possibility to deal with a large variety of graphs (with no information/discrete/continuous attached to nodes). Not sure about the impact on cases where information is attached to nodes. Follow-up could involve other ways to code structural information.  After rebuttal. I am not convinced about the argument with tottering. Even if labels are not considered, still there will be a strong bias on closer nodes. The authors should find a more convincing argument.


Dynamic Network Model from Partial Observations
Can evolving networks be inferred and modeled without directly observing their nodes and edges? In many applications, the edges of a dynamic network might not be observed, but one can observe the dynamics of stochastic cascading processes (e.g., information diffusion, virus propagation) occurring over the unobserved network. While there have been efforts to infer networks based on such data, providing a generative probabilistic model that is able to identify the underlying time-varying network remains an open question. Here we consider the problem of inferring generative dynamic network models based on network cascade diffusion data. We propose a novel framework for providing a non-parametric dynamic network model-based on a mixture of coupled hierarchical Dirichlet processesbased on data capturing cascade node infection times. Our approach allows us to infer the evolving community structure in networks and to obtain an explicit predictive distribution over the edges of the underlying network-including those that were not involved in transmission of any cascade, or are likely to appear in the future. We show the effectiveness of our approach using extensive experiments on synthetic as well as real-world networks.
Existing models, however, aim to model a fully observed network [4,5,8,9] but in many realworld problems, the underlying network structure is not known. What is often known are partial observations of a stochastic cascading process that is spreading over the network. A cascade is created by a contagion (e.g., a social media post, a virus) that starts at some node of the network and then spreads like an epidemic from node to node over the edges of the underlying (unobserved) network. The observations are often in the form of the times when different nodes get infected by different contagions. A fundamental problem, therefore, is to infer the underlying network structure from these partial observations. In recent years, there has been a body of research on inferring diffusion networks from node infection times. However, these efforts mostly rely on a fixed cascade transmission model-describing how nodes spread contagions-to infer the set of most likely edges [2,10,11,12]. More recently, there have been attempts to predict the transmission probabilities from infection times, either by learning node representations [13], or by learning diffusion representations using the underlying network structure [13,14,15]. However, it remains an open problem to provide a generative probabilistic model for the underlying network from partial observations.
Here we propose a novel online dynamic network inference framework, DYFERENCE, for providing non-parametric edge-exchangeable network models from partial observations. We build upon the non-parametric network model of [8], namely MDND, that assumes that the network clusters into groups and then places a mixture of Dirichlet processes over the outgoing and incoming edges in each cluster while coupling the network using a shared discrete base measure. However, our framework is easily extended to arbitrary generative models replacing the MDND with other choices of latent representations, such as network models presented in [9,16,17,18]. Given a set of cascades spreading over the network, we process observations in time intervals. For each time interval we first find a probability distribution over the cascade diffusion trees that may have been involved in each cascade. We then calculate the marginal probabilities for all the edges involved in the diffusion trees. Finally, we sample a set of edges from this distribution and provide the sampled edges to a Gibbs sampler to update the model variables. In the next iteration, we use the updated edge probabilities provided by the model to update the probability distributions over edges supported by each cascade. We continue the above iterative process until the model does not change considerably. Extensive experiments on synthetic and real-world networks show that DYFERENCE is able to track changes in the structure of dynamic networks and provides accurate online estimates of the time-varying edge probabilities for different network topologies. We also apply DYFERENCE for diffusion prediction and predicting the most influential nodes in Twitter and MemeTracker datasets, as well as bankruptcy prediction in a financial transaction network.

The authors propose a framework for modeling dynamic network cascade diffusion and contrast the approach in particular to InfoPath but also several more recent static network estimation procedures. The model is probabilistic and extends the work of Williams (ref 7) to cascade diffusion by learning the latent network structure solely based on the nodes observed contamination times within a cascade. The framework uses the partial observational framework discussed in ref 27 and inference of determinantal point processes as discussed in ref 28 in order to establish a distribution of plausible edges given the observed cascades. The MDMN model of ref 7 is then used as a latent representation of this unobserved network representation. The framework is extended to the modeling of dynamic networks considering a time interval w and updating the model at time t+1 using the model at time t as a starting point.   Quality: The manuscript is well written, the experimentation solid, and the approach well motivated improving upon current state-of-the-art of network quantification based on observed cascades.  Clarity: The manuscript is generally well written and well rooted in the existing literature. There are a few typos that should be corrected, i.e. “is the |Rk x Rk| is the restriction…” -> “is the |Rk x Rk| restriction…” “and and” -> “and” I would further recommend that the section “Related Work” be integrated into the existing introduction as there are quite some overlaps between the introduction and related work section. In particular, remarks such as “Our work is different in nature to the existing methods in that we aim at providing a generative probabilistic  model for the underlying dynamic network from diffusion data.” and “However, these models assume that networks are fully observed. In contrast, our work here considers the network is unobserved but what we observe are node infection times of a stochastic cascading process spreading over the network.” These points are already remarked in the introduction also discussing some of the current state of the art. It could on related work also be worth mentioning probabilistic generative dynamic models assuming fully observed networks, see for instance as part of the related work Ishiguro, Katsuhiko, et al. "Dynamic infinite relational model for time-varying relational data analysis." Advances in Neural Information Processing Systems. 2010.  Originality: The novelty of the paper is combining the use of latent network modeling based on the mixture of   Dirichlet network distributions (MDND) modeling framework ref 7 to operate not directly on an observed network but a latent network defined though cascades using results from ref. 27 and 28 in order to define a distribution over latent edges. In ref 7 the MDND was established as a competitive link-prediction framework and it is thus reasonable to use this as a starting point of latent variable modeling of the network structure. However, other competitive probabilistic link prediction modeling approaches choices could also have been relevant choices, such as:  Palla, Konstantina, David Knowles, and Zoubin Ghahramani. "An infinite latent attribute model for network data." arXiv preprint arXiv:1206.6416 (2012). And within the framework of Fox and Carron the extension to modeling block structure Herlau, Tue, Mikkel N. Schmidt, and Morten Mørup. "Completely random measures for modelling block-structured sparse networks." Advances in Neural Information Processing Systems. 2016. It would improve the paper to discuss how the present framework extends to arbitrary generative models replacing the MDND with other choices of latent representations which would expand the scope of the present framework. That being said there are many merits of the MDND and this is a sound and strong starting point. Combining generative network models for modeling cascades is an interesting and useful novel contribution.  Significance: The proposed framework is a fine extension of the MDND framework to the modeling of network structure based on observed cascades and form a new and powerful tool for analyzing network dynamics. In particular, the experimental results emphasizes the utility of the proposed framework outperforming existing approaches while providing a nice probabilistic generative modeling framework for the extraction of dynamic network structure.  It is unclear to me why the network needs to be discretized in time segments w as the cascades are in continuous time. Furthermore, the dynamic modeling stemming from updating the model from previous timesteps seem a bit heuristic and to hinge on how much the sampling procedure changes the model across the time steps which in turn also reflect the ability of the sampler to properly mix and for how long the inference procedure is run (i.e., how many samples are drawn at each time step). It would improve the manuscript to clarify these aspects. It would also be good to include a discussion of the stability of the inferred parameters and thus reliability of the inference procedure. Furthermore, it would be good to clarify how the predictions are made, i.e. InfoPath rely on point estimates but the current procedure I assume is based on averaging samples. This should be clarified, and if averaging samples it would be good to also demonstrate what the performance of a corresponding point estimate (by highest likelihood sample) would provide to assess the merits of simply providing Bayesian averaging as compared to a more advanced MDND model.  I have read the authors rebuttal and find that they satisfactorily discuss the raised concerns. I argue for accepting this paper.