MetaGAN: An Adversarial Approach to Few-Shot Learning
In this paper, we propose a conceptually simple and general framework called MetaGAN for few-shot learning problems. Most state-of-the-art few-shot classification models can be integrated with MetaGAN in a principled and straightforward way. By introducing an adversarial generator conditioned on tasks, we augment vanilla few-shot classification models with the ability to discriminate between real and fake data. We argue that this GAN-based approach can help few-shot classifiers to learn sharper decision boundary, which could generalize better. We show that with our MetaGAN framework, we can extend supervised few-shot learning models to naturally cope with unlabeled data. Different from previous work in semi-supervised few-shot learning, our algorithms can deal with semi-supervision at both sample-level and task-level. We give theoretical justifications of the strength of MetaGAN, and validate the effectiveness of MetaGAN on challenging few-shot image classification benchmarks.

This paper proposes a method of improving upon existing meta-learning approaches by augmenting the training with a GAN setup. The basic idea has been explored in the context of semi-supervised learning:   add an additional class to the classifier's outputs and train the classifier/discriminator to classify generated data as this additional fake class. This paper extends the reasoning for why it might work for semi supervised learning to why is might work for few-shot meta learning.   The clarity of this paper could be greatly improved. They are presenting many different variants of few-shot learning in supervised and semi-supervised setting, and the notation is a bit tricky to follow initially. The idea proposed in this paper is a very simple one, yet the paper was a bit laborious to read.  Overall, this paper on the border of acceptance.  The idea is simple, and although it has been used in other contexts, its application in this area is novel. However, since the basic ideas in this paper a not new, and rather just a combination of existing work, I would hope for a bit more empirical evaluations (for example, in the RL setting as MAML is applied). That said, the method is simple and improves upon existing approaches and so I am (very marginally) on the positive side of acceptance boundary, provided the paper could edited for clarity.   Detailed comments: - The notation in "few shot learning section" is a bit hard to read. For example, the support set is given as S_T = S^s_T U S^u_T and then S^s_T is described as the supervised support set, so I would assume S^u_T is the unsupervised support set. But then U_T is defined as the unlabelled support set , so what is S^u_T? Also, what do the *'s on line 71 mean? Why are some of the Y's *-ed and others not? In general, this paragraph could use some editing.  - In table 2, why do you only apply the metaGAN framework to MAML with 1 step, first order? Why not the other versions, which are better than that one? - how significant are the omniglot improvements. Most gains look to be within error bars. - could the metaGAN approach be combed with the k-means method in table 3 in order to improve upon it?


On Coresets for Logistic Regression
Coresets are one of the central methods to facilitate the analysis of large data. We continue a recent line of research applying the theory of coresets to logistic regression. First, we show the negative result that no strongly sublinear sized coresets exist for logistic regression. To deal with intractable worst-case instances we introduce a complexity measure µ(X), which quantifies the hardness of compressing a data set for logistic regression. µ(X) has an intuitive statistical interpretation that may be of independent interest. For data sets with bounded µ(X)-complexity, we show that a novel sensitivity sampling scheme produces the first provably sublinear (1 ± ε)-coreset. We illustrate the performance of our method by comparing to uniform sampling as well as to state of the art methods in the area. The experiments are conducted on real world benchmark data for logistic regression.

The goal of this paper is to speed up logistic regression using a coreset based approach. The key idea is to “compress” the data set into a small fake set of points (called coreset) and to then train on that small set. The authors first show that, in general, no sublinear size coreset can exist. Then, they provide an algorithm that provides small summaries for certain data sets that satisfy a complexity assumption. Finally, they empirically compare that algorithm to two competing methods.   Overall, the paper seems sound on the theoretical side; however, there are weaknesses in the related work and the experiments. If some of these concerns would be addressed in the rebuttal, I would be willing to upgrade my recommended score.  Strengths:  - The results seem to be correct.  - In contrast to Huggins et al. (2016) and Tolochinsky & Feldman (2018), the coreset guarantee applies to the standard loss function of logistic regression and not to variations.  - The (theoretical) algorithm (without the sketching algorithm for the QR decomposition) seems simple and practical. If space permits, the authors might consider explicitly specifying the algorithm in pseudo-code (so that practitioners do not have to extract it from the Theorems).  - The authors include in the Supplementary Materials an example where uniform sampling fails even if the complexity parameter mu is bounded.   - The authors show that the proposed approach obtains a better trade-off between error and absolute running time than uniform sampling and the approach by Huggins et al. (2016).    Weaknesses:  - Novelty: The sensitivity bound in this paper seems very similar to the one presented in [1] which is not cited in the manuscript. The paper [1] also uses a mix between sampling according to the data point weights and the l2-sampling with regards to the mean of the data to bound the sensitivity and then do importance sampling. Clearly, this paper treats a different problem (logistic regression vs k-means clustering) and has differences. However, this submission would be strengthened if the proposed approach would be compared to the one in [1]. In particular, I wonder if the idea of both additive and multiplicative errors in [1] could be applied in this paper (instead of restricting mu-complexity) to arrive at a coreset construction that does not require any assumptions on the data data set.  [1] Scalable k-Means Clustering via Lightweight Coresets Olivier Bachem, Mario Lucic and Andreas Krause To Appear In International Conference on Knowledge Discovery and Data Mining (KDD), 2018.  - Practical significance: The paper only contains a limited set of experiments, i.e., few data sets and no comparison to Tolochinsky & Feldman (2018). Furthermore, the paper does not compare against any non-coreset based approaches, e.g., SGD, SDCA, SAGA, and friends. It is not clear whether the proposed approach is useful in practice compared to these approaches.  - Figure 1 would be much stronger if there were error bars and/or if there were more random trials that would (potentially) get rid of some of the (most likely) random fluctuations in the results.


An Information-Theoretic Analysis for Thompson Sampling with Many Actions
Information-theoretic Bayesian regret bounds of Russo and Van Roy [8]  capture the dependence of regret on prior uncertainty. However, this dependence is through entropy, which can become arbitrarily large as the number of actions increases. We establish new bounds that depend instead on a notion of rate-distortion. Among other things, this allows us to recover through information-theoretic arguments a near-optimal bound for the linear bandit. We also offer a bound for the logistic bandit that dramatically improves on the best previously available, though this bound depends on an information-theoretic statistic that we have only been able to quantify via computation.

POST REBUTTAL:  Thanks for the response. My scores stay the same. Of course I encourage the authors to try and relax the assumptions for the generalized linear models and prove the conjecture.  Summary.  The paper provides a Bayesian regret-analysis of Thompson sampling for linear bandits with large action spaces. Previous bounds depended on the number of actions, which is known to be suboptimal when the number of actions is super-exponential in the dimension. Existing Bayesian analysis for this case relied on the conversion from frequentist bounds, which is (a) excessively complicated and (b) leads to looser bounds.  The authors give a simple and rather intuitive analysis based on the observation that the learner only needs to learn the unknown parameter up to some epsilon and so the entropy of the unknown parameter should be replaced by the entropy of a partitioning random variable.  The new analysis is applied to the standard linear bandit model where O(d sqrt(T log(T))) bounds are provided that improve on the state-of-the-art by a factor of sqrt(log(T)). The bounds are also applied to generalized linear bandits. More comments on this to follow.  Major comments.  I think this is a nice paper. Some questions/comments. First, there is an assumption that the rewards are bounded and this seems quite important for the proofs. For the standard frequentist analysis, however, one assumes the rewards are subgaussian. Of course you can apply a union bound over all the observed rewards, but I think a log(T) factor in the regret will sneak in if you do this. Do you know how to relax the assumption on the noise to (say) conditionally subgaussian?  Assumptions 1 and 3 both seem quite unreasonable. In applications of GLMs that I know of the rewards are almost always Bernoulli and the expected rewards are almost always very small. Eg., the probability that a user clicks on something. Can you say more about whether or not these assumptions can be relaxed?  Clarity.  The paper is mostly well written. I think some discussion of the assumptions is appropriate and there are a few typos (see below).  Quality.  The proofs seem to be correct. The problem is interesting.  Originality.  This is the weakest part of the paper. The idea to use a discretization is quite an obvious one and the analysis does not change that much from previous work in this area.  Significance.  I think there will be plenty of interest in this paper. The paper would be more significant if the assumptions for generalized linear models were more realistic.  Overall.  A good paper. Perhaps the authors 'stopped early' because there are lots of unresolved questions and the boat is not pushed too much further. Still, I prefer to see it accepted.  Minors.  * Is \mathcal Y finite? Measurability of g? Almost sure qualifications on conditional expectations.  * Why not mention in (1) that I(. ; . | . ) is the conditional mutual information?  * Both H_t and ~H_t generate different sigma-algebras, but the extra information in ~H_t does not change the conditional measure (which is just the posterior - though this is not unique... measurability, brr). Anyways, is there some subtle reason behind this change or do you just want to emphasize the view of sampling thetas?  * In (d) of Eq. (6) should ~H_T be ~H_{T-1}?  * L137. p^* should be theta^*?  * You could be consistent about the order of theta^* and psi in (6) and the display after, though of course it is symmetric.  * L158. \in -> \subset  * L160. Did you mean to say that the action set and parameter space or dual to each other? Otherwise I don't see where you actually assume the inner product is bounded. The 'normalizing factor' of 1/2 is nonstandard. Why not just let mu(a, theta) =  and assume || <= 1/2 for all a, theta?  * Assumption 1 is quite restrictive. One reason to introduce GLMs in the first place is to behave well with Bernoulli noise.  * Assumption 3 is also quite restrictive.


Meta-Learning MCMC Proposals
Effective implementations of sampling-based probabilistic inference often require manually constructed, model-specific proposals. Inspired by recent progresses in meta-learning for training learning agents that can generalize to unseen environments, we propose a meta-learning approach to building effective and generalizable MCMC proposals. We parametrize the proposal as a neural network to provide fast approximations to block Gibbs conditionals. The learned neural proposals generalize to occurrences of common structural motifs across different models, allowing for the construction of a library of learned inference primitives that can accelerate inference on unseen models with no model-specific training required. We explore several applications including open-universe Gaussian mixture models, in which our learned proposals outperform a hand-tuned sampler, and a real-world named entity recognition task, in which our sampler yields higher final F1 scores than classical single-site Gibbs sampling.

This paper describes a method to learn smart proposals for speeding up MCMC inference in probabilistic graphical models (PGMs). The main idea is to decompose the graph into possibly overlapping "structural motifs" such as grids or chains with similar relationships between the variables that are either proposed variables, conditioned variables, or local parameters. Using an appropriate distribution, samples are generated for training a probability distribution on the proposed variables given the conditioned variables and local parameters for each instantiation of the motif in the PGM. These learned proposal distributions are then used to generate proposals which are then accepted/rejected using the MH rule. (Variables not covered by any motif are sampled using single-site Gibbs).  This work is very similar to inference compilation cited as [19] in the paper. However, the big difference is that inference compilation works on training a proposer for the entire probabilistic program whereas this paper builds up the proposer from multiple parts where each part is trained on a smaller sub-structure in the probabilistic model. The hope being that training a proposer on smaller sub-structures would allow those to be reused across models.  Overall the work is quite sound and the results are very promising. The main reason for my low overall rating is to do with the disparity between the claims made in the abstract/introduction with the actual contents of the text. The authors claim, for example, that the "The learned neural proposals generalize to occurrences of common structural motifs across different models, allowing for the construction of a library of learned inference primitives that can accelerate inference on unseen models with no model-specific training required." However, there is no evidence given in the text of a proposer being trained on one model and then used for inference on another one.  The other drawback of the paper is that the authors seem to suggest that they have a magical blackbox method that works in all models without any hand tuning as opposed to other techniques (line 51). But in every experiment described their appears to be a fair amount of hand tuning in terms of figuring out how to generate the data for training the proposers. In one case they decide to collapse out a large number of the variables all together! This is more of a collection of recipes rather than a systematic procedure for training the proposers.   Finally, it appears that the work is not applicable beyond the realm of fixed size models for two reasons. 1) Even though the authors mention probabilistic programming languages (PPLs) early in the introduction there is no follow up in the text. For example, one would expect some discussion about extracting motifs in the context of a concrete PPL. Also, the paper assumes that the motifs can be extracted at the beginning of inference (Algorithm 1). However, in PPLs the graph structure changes with different values of the latent variables, so it doesn't appear applicable. 2) I don't buy the open universe probabilistic model (OUPM) example in the text. Instead of sampling the number of objects in the universe either directly or lazily (through a Dirichlet process) this example has an upper limit on the number of objects and assigns an "activity" indicator to each possible object with the total number of objects defined as the sum of the active objects. This is an oversimplification. OUPMs are meant to capture not just an unknown number of objects, but also potentially unbounded number of objects. There is nothing in the text of this paper that describes how a proposer would intelligently determine adding or subtracting objects from the model.  Despite these drawbacks I believe that the work describes a significant improvement over single-site Gibbs for fixed size models with discrete-valued variables, and is worth publishing.   More detailed comments:  - The toy example in Fig 1 is misleading. It seems to suggest that the Neural Network is trained only on the local model parameters alpha and beta. While in reality it appears that the neural network is trained on the local model parameters as well as the conditioning variables.  - It is truly baffling that the conditioning variables for a motif might not include the Markov Blanket of the proposed variables. This appears to run contrary to the theme of the paper that the proposers learned from these motifs are reusable. For example this work is attempting to approximate the probability distribution p_\psi_i(B_i|C_i) at the beginning of inference, but this distribution depends on the values of the remaining variables in MB(B_i) - C_i which can change as inference progresses. In other words, it appears that the proposal that is learned may not even be relevant from sample to sample let alone be reused across models. The GMM is the only example in the experiments where the Markov Blanket is not a subset of the conditioning set. In the GMM case I suspect that the n=60 randomly selected samples that are part of the motif are a good enough approximation of the Markov Blanket. It would have been helpful to demonstrate more clearly with an example why the authors decided that C_i doesn't equal MB(B_i).  - Line 91 talks about "training with random model parameterization", but this is not a well developed concept in this paper. There are a series of examples in the experiments, but no crisp procedure.  - Line 208, 209. It is a stretch to conclude from one observed low KL-divergence in this small grid example that this approach works in general to approximate true conditionals. Also, it is not clear from the text if the inference used a single proposal for all occurrences of the grid motif in the graph or were separate proposers trained for each grid motif. The other unanswered question is the degree of overlap between grids, and whether that affects inference.  Equation (4) is a bit confusing. This is describing a distribution for a binary valued variable, right? So why do we have "[0,1]" on the first line and "[1,0]" on the second line instead of say "0" on the first line and "1" on the second line?  - Line 252. It is not entirely clear how the samples for training the proposer were obtained. Is the model being simulated to generate samples for the training data (as in [19])?   Minor:  Line 162. I am assuming that a probability table for a bounded discrete table is simply a categorical distribution over the possible values. So, then why do we need a mixture of these. Can't a mixture of categoricals be represented as a single categorical?  1 (a) -> om -> on  Post-Response:  I am willing to accept the authors response that collapsing out the GMM is not necessary to get better performance, and that they will provide results without collapsing in the next version. However, the claim that OUPMs can be handled with RNNs requires more scrutiny, and such claims should not be added without review. For the current NIPS'18 submission, the discussion on OUPMs and PPLs properly belongs in the "future work" section of the paper given the sketchy details.  The paper still has a number of important under-developed concepts such as not using the Markov Blanket for the conditioning set. The authors haven't explained why this choice of the conditioning set doesn't affect reusability of the trained proposers.


Active Matting
Image matting is an ill-posed problem. It requires a user input trimap or some strokes to obtain an alpha matte of the foreground object. A fine user input is essential to obtain a good result, which is either time consuming or suitable for experienced users who know where to place the strokes. In this paper, we explore the intrinsic relationship between the user input and the matting algorithm to address the problem of where and when the user should provide the input. Our aim is to discover the most informative sequence of regions for user input in order to produce a good alpha matte with minimum labeling efforts. To this end, we propose an active matting method with recurrent reinforcement learning. The proposed framework involves human in the loop by sequentially detecting informative regions for trivial human judgement. Comparing to traditional matting algorithms, the proposed framework requires much less efforts, and can produce satisfactory results with just 10 regions. Through extensive experiments, we show that the proposed model reduces user efforts significantly and achieves comparable performance to dense trimaps in a user-friendly manner. We further show that the learned informative knowledge can be generalized across different matting algorithms.

Due to the importance of the user input trimap or strokes in image matting, this paper propose an active matting method with recurrent reinforcement learning. The proposed model can actively suggest a small number of informative sequence of regions for user to label, which minimizes the labeling efforts. Experimental results show that the proposed model achieves comparable performance to dense trimaps. In addition, the learned informative knowledge can be applied to a variety of matting algorithms as shown in this paper.  Strengths: 1. This paper proposes a novel an helpful concept named informative regions to denote the regions that affect the matting performance most.  2. The proposed model is in a user-friendly manner that users are only required to label a suggested region as foreground or background. 3. Extensive results show that the learned informative regions achieve comparable matting performance to a fine-labelled trimap in terms of quality while the proposed method is much faster. 4. The proposed model have well applicability to a variety of matting algorithms. 5. The details are well clarified and the method can be reproduced according to this paper. Weaknesses: 1. The authors run their network S times repeatedly and collect S latent regions as suggestions each iteration. They set S to 2 to achieve a good balance between accuracy and efficiency. More details are missing to support this choice. 2. Several important references are missing. For example, 1) Ning Xu, Brian Price, Scott Cohen, Thomas Huang, Deep Image Matting. CVPR 2017. the authors are encouraged to add comparisons.  2) Liang Yang, Xiaochun Cao, Di Jin, Xiao Wang, Dan Meng: A Unified Semi-Supervised Community Detection Framework Using Latent Space Graph Regularization. IEEE Trans. Cybernetics 45(11): 2585-2598 (2015) where a similar "active" must-link priors are utilized.  3) Chuan Wang, Hua Zhang, Liang Yang, Xiaochun Cao, Hongkai Xiong: Multiple Semantic Matching on Augmented N-Partite Graph for Object Co-Segmentation. IEEE Trans. Image Processing 26(12): 5825-5839 (2017) where co-matting is addressed. 3. In my point of view, the most important significance of this paper is that is can help establishing large-scale matting dataset, which is rarely mentioned in this paper.


Bounded-Loss Private Prediction Markets
Prior work has investigated variations of prediction markets that preserve participants' (differential) privacy, which formed the basis of useful mechanisms for purchasing data for machine learning objectives. Such markets required potentially unlimited financial subsidy, however, making them impractical. In this work, we design an adaptively-growing prediction market with a bounded financial subsidy, while achieving privacy, incentives to produce accurate predictions, and precision in the sense that market prices are not heavily impacted by the added privacy-preserving noise. We briefly discuss how our mechanism can extend to the data-purchasing setting, and its relationship to traditional learning algorithms.

This paper studies prediction markets: mechanisms where a series of buyers arrive and make bets on the outcome of a future event, where the prices of bets at a certain point of time reflect the information gathered so far about this outcome. Previous work by Waggoner et al studies differentially-private prediction markets, where no individual buyer's trade is inferrable from the overall sequence of market prices (this is accomplished by adding some amount of noise to each of the revealed market states). These markets have the downside that the expected cost to the mechanism designer of maintaining this differential privacy is unbounded, growing perhaps linearly in T (number of rounds/buyers). This paper extends the previous work by proposing new mechanisms that have bounded total cost to the mechanism designer. Specifically, the authors:  1. The authors show that adding an appropriately chosen transaction fee to the mechanism of Waggoner et al causes the total loss of the mechanism to be bounded by O(log^2 T). (Intuitively, the noise added to maintain differential privacy creates a small arbitrage opportunity for each of the buyers; adding a transaction fee on that scale eliminates this).  2. The authors then show how to "amplify" this O(log^2 T)-loss mechanism to construct a mechanism with O(1)-total loss. This works in a way reminiscent of the doubling trick -- after enough buyers have participated in the current mechanism, the mechanism fee restarts with different parameters (i.e. half the transaction fee).   I am not very familiar with the peer prediction literature (including the previous paper by Waggoner et al which this paper builds off of significantly), but this seems like a very strong result, addressing perhaps the largest problem with the previous construction for differentially private prediction markets. In fact, it is not at all clear a priori that a differentially private prediction market with loss bounded by a constant necessarily exists (indeed, Cummings et al show that any of a large class of differentially private prediction markets -- ones that just add some amount of noise to the market states -- must necessarily have unbounded loss).  The paper is well-written and easy to read. The proofs in the main body of the paper appear correct (although I have not checked the proofs in the appendix).


Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base
We present an approach to map utterances in conversation to logical forms, which will be executed on a large-scale knowledge base. To handle enormous ellipsis phenomena in conversation, we introduce dialog memory management to manipulate historical entities, predicates, and logical forms when inferring the logical form of current utterances. Dialog memory management is embodied in a generative model, in which a logical form is interpreted in a top-down manner following a small and flexible grammar. We learn the model from denotations without explicit annotation of logical forms, and evaluate it on a large-scale dataset consisting of 200K dialogs over 12.8M entities. Results verify the benefits of modeling dialog memory, and show that our semantic parsing-based approach outperforms a memory network based encoder-decoder model by a huge margin.

This paper proposes a sequence-to-sequence model to tackle the conversational knowledge based QA. Unlike existing seq2seq models, they additionally incorporate a conversation memory into the model, which enables the model to copy previous action subsequences. Experimental results show that this model significantly outperforms the KVMem based model. The paper is well written.  I have a few concerns about this paper:  (1) There is a basic assumption throughout the paper, that is, a logical form could be generated using the grammar.     This requires that the logical form could be represented as a tree. However, I do think this assumption always holds given the fact that     many meaning representations are represented in more complicated structure. For example, the Abstract Meaning Representation (AMR) is represented in a graph structure.  (2) It is very hard to follow in the section 6.2. Many symbols in the equation are not described. For example, s_{t} and a_{
Reviewer 2
This paper proposes a semantic parsing method for dialog-based QA over a large-scale knowledge base. The method significantly outperforms the existing state of the art on CSQA, a recently-released conversational QA dataset. One of the major novelties of this paper is breaking apart the logical forms in the dialog history into smaller subsequences, any of which can be copied over into the logical form for the current question. While I do have some concerns with the method and the writing (detailed below), overall I liked this paper and I think that some of the ideas within it could be useful more broadly for QA researchers.   Detailed comments: - I found many parts of the paper to be confusing, requiring multiple reads to fully understand. Section 4, for example, describes a "dialog memory", but it is unclear whether and how any of the subcomponents (e.g., predicates, actions) are represented by the model until much later. The paper could really benefit from a concise summary of the method before Section 4, as currently readers are just thrown into details with no context. - The inclusion of actions in the dialog memory is very interesting! This could benefit models trained on other sequential QA datasets as well and strikes me as the most unique part of this paper.  - Q2 in Figure 2 ("Where did he graduate from?") is different than the Q2 referenced in the text ("Where is its capital?"), and the figure appears to contain the logical form for the latter question.  - The context here is just the previous question. It would be nice if the authors would describe the challenges associated with scaling the approach to the entire dialog history, and perhaps offer a small-scale analysis of the impact of increasing the context size. - The implementation of the "replication actions" are a bit confusing. From my understanding, the decoder first decides to choose a replication action (e.g., A19), and only then decides what entities/predicates/numbers from the dialog memory to replicate. Was this done for efficiency reasons? ICould all of the "action subsequences" be enumerated and then added as separate question-specific actions? - What is the purpose of the "gates" in the replication actions (Sec 6.2)? What does it mean for a gate to be chosen? Don't you already know beforehand where the entities/etc. in the memory come from? Relatedly, why is there only one gate for numbers? Surely numbers can occur in the previous question and previous response... - It would be nice to have some discussion of how time-intensive the learning process detailed in 6.3 / Appendix B is. As the search space is huge, doing BFS is likely very expensive! - Since the dataset is new and the comparisons are only against baselines in the original dataset paper, it's hard to be overly impressed with the huge performance gains. That said, the different ablated versions of the proposed model make up for this and offer nice insights into the relative importance of the different components in the dialog memory.  - Section 7.3 is much appreciated; I'd like to see a bit more in the "Unsupported Actions" section about questions that require reasoning across multiple questions.
Reviewer 3
Update:  Thank you authors for your response.   1) My original comments regarding the lack of references is misleading. What I meant was that I think the authors should immediately (e.g. in the first paragraph) frame this work in the context of prior work, say, in semantic parsing. The introduction, as currently written, does not provide good grounding for this paper.   2) I think you should explicitly define what you mean by "consistent" such that the reader understands this claim.  4) I am now able to access the website for the dataset (it was down when I submitted my review). In future papers, please use the citation for the peer reviewed paper instead of the Arxiv version. This is especially important for work as crucial as the task your paper is working on.  Thanks for your response. I have increased my score.    Original text:  Even though the authors do not mention semantic parsing in the introduction, It seems to me that the authors proposed a semantic parser (e.g. one that maps utterances to logical forms). This parser does not use logical forms directly as supervision and rather learns from denotations (http://nlp.stanford.edu/pubs/pasupat2016inferring.pdf). This work aims to resolve coreference resolution in multi-turn semantic parsing by examining conversation history.  I think the writing needs more polish. In particular, Figure 1 lacks logical forms, which makes the references in the introduction difficult to follow. Moreover, the authors should just use accepted vocabulary instead of inventing their own (e.g. coreference resolution, though I may have misunderstood the motivation). The notation in section 4 and Figure 2 are not introduced. It is not clear what find(set, r1) means until section 5, despite the fact that this understanding is crucial in comprehension of the proposed method.   In section 6.2: - it is not clear to me what a subsequence is - it would help if the authors gave an example - What is v_sub? Is it the GRU encoding of sub? - On line 170, what do you mean by "consistent"?  The objective function described in 6.3 seems expensive and intractable for larger problems. Moreover, it seems highly prone to false positives from spurious logical forms.  The choice of evaluation is not ideal. This particular task seems to be un-refereed. I was also not able to access it at the time of review (e.g. the website was down), nor was I able to find refereed, related works on this task. I would be more convinced by the results if the authors either compare to previously refereed work or demonstrate the effectiveness of this work on an established, refereed task.  Minor feedback: - I would refrain from high subjective adjectives such as "huge margin", "outperforms dramatically", etc. and leave the judgement to the reader.


Information Constraints on Auto-Encoding Variational Bayes
Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on auto-encoding variational Bayes, in which the search space is constrained via kernel-based measures of independence. In particular, our method employs the d-variable Hilbert-Schmidt Independence Criterion (dHSIC) to enforce independence between the latent representations and arbitrary nuisance factors. We show how this method can be applied to a range of problems, including problems that involve learning invariant and conditionally independent representations. We also present a full-fledged application to singlecell RNA sequencing (scRNA-seq). In this setting the biological signal is mixed in complex ways with sequencing errors and sampling effects. We show that our method outperforms the state-of-the-art approach in this domain.

In this work, the authors propose an approach to enforce independence among components of latent representations learned with AEVB. This amounts to adding a penalty term based on the Hilbert-Schmidt Independence Criterion to the ELBO. A set of experiments suggest the approach learns independent components.  The problem is well-motivated, and theoretic justification is given for why the approach could be expected to work. The experiments are also reasonable, but the lack of interpretation of the results on the real datasets makes it difficult to tell whether the results are in some sense “good.” There is also little theoretical comparison between the proposed approach and similar (cited) work.  === Major comments  As mentioned above, the lack of explanation of the experimental results makes it difficult to interpret them. For example, in Table 1, the *worst* results for the lighting group and direction are in bold. Presumably, that is actually good because it means the latent representations are ignoring the nuisance (lighting) information. However, a more clear explanation would make this point more obvious. Similarly, it is not obvious what the evaluation metrics used on the sc-seq data really show.  The main difference compared to \beta-TCVAE and VFAE seems to be the choice of mutual information measure. Thus, it would be helpful to more explicitly compare dHSIC and the measures used in the other approaches. Otherwise, it is not clear why we would expect the proposed approach to outperform those.  === Minor comments  At the end of Section 3.1, the authors propound that “the independence properties… will often not be respected by the approximate posterior.” While I agree with this, a reference or two which either prove or demonstrate this empirically would be helpful.  Figure 2 should also show results for an unconstrained VAE as a baseline.  For reference, Table 1 should include unconstrained VAE results. This would make more clear what is “lost” (in terms of accuracy) by encouraging independence among the latent features.   === Typos, etc.  “equation 5” -> “Equation 5”  The “superscript” notation (presumably indicating a single dimension/variable in the respective space) is not defined.  The fonts on Figure 2 should be increased. Also, it is not understandable when printed in black and white. Different marker types could fix that.  “has be shown” -> “has been shown”  MMD is not defined.  “Dirichlet kernel” in Proposition 2. This seems more like the definition of the Dirac delta function.  The references are inconsistently formatted. Also, some of the names are misspelled, like “Schlkopf”.  --- After author feedback  The authors have addressed my concerns.


Data Amplification: A Unified and Competitive Approach to Property Estimation
Estimating properties of discrete distributions is a fundamental problem in statistical learning. We design the first unified, linear-time, competitive, property estimator that for a wide class of properties and for all underlying distributions uses just 2n samples to achieve the performance attained by the empirical estimator with n √ log n samples. This provides off-the-shelf, distribution-independent, "amplification" of the amount of data available relative to common-practice estimators. We illustrate the estimator's practical advantages by comparing it to existing estimators for a wide variety of properties and distributions. In most cases, its performance with n samples is even as good as that of the empirical estimator with n log n samples, and for essentially all properties, its performance is comparable to that of the best existing estimator designed specifically for that property.

This paper considers estimation of functionals of distributions in the high-dimensional regime, and designs an estimator that “amplifies” the amount of data required to attain the same error as compared to the plug-in approach.  In particular, the least squares error of functional estimation employing order n\sqrt{\log n} samples to compute the plug-in estimator can be matched by their estimator in just order n samples. The estimator uses a variant of the plug-in estimator for higher probabilities, and a polynomial smoothing technique for low probabilities, with a data-dependent threshold to decide whether a particular symbol probability is low or high.   The estimator runs in inear-time in number of samples, is competitive with the plug-in estimator with a larger number of samples, simple to compute, and sees improvement in simulations. The rates improve upon the plug-in estimator in the high dimensional regime where the support size is greater than the number of samples.  The paper is well-written and easy to follow, and I liked reading it. However, I did have a few concerns with some parts of it, in particular about how significant such an amplification result really is. These are listed in detail below:  Concerns:  1. As the authors mention is Section 4, Theorem 1 only really provides improvement over the least squares estimate when the loss of empirical (plug-in) estimator is slowly decaying ( log^{-\eps} n). In particular, this means that even with data amplification and substituting n for n\sqrt{n}, the rate of the improved estimator can only be improved in the second (double logarithmic) term. While such slow rates are understandable in high dimensions, the afforded improvement seems miniscule here.  2. As a counter to the point above, the authors show that for some functionals, the rate of the plug-in estimate beyond which one gains improvement can actually be as low as log^{-2} n. In this case, too, it seems like the amplification afforded would only be doubly logarithmic in the rate.  3. While the two points are not major weaknesses (the simulations seem to address whether or not an improvement is actually seen in practice), the paper would benefit from a discussion of the fact that the targeted improvement is in the (relatively) small n regime.  4. The paper would benefit from a more detailed comparison with related work, in particular making a detailed comparison to the time complexity and competitiveness of prior art.  Minor: 1. The proofs repeatedly refer to the Cauchy inequality, but it might be better given audience familiarity to refer to it as the Cauchy-Schwarz inequality.  Post-rebuttal: I have read the authors' response and am satisfied with it. I maintain my vote for acceptance.


Link Prediction Based on Graph Neural Networks
Link prediction is a key problem for network-structured data. Link prediction heuristics use some score functions, such as common neighbors and Katz index, to measure the likelihood of links. They have obtained wide practical uses due to their simplicity, interpretability, and for some of them, scalability. However, every heuristic has a strong assumption on when two nodes are likely to link, which limits their effectiveness on networks where these assumptions fail. In this regard, a more reasonable way should be learning a suitable heuristic from a given network instead of using predefined ones. By extracting a local subgraph around each target link, we aim to learn a function mapping the subgraph patterns to link existence, thus automatically learning a "heuristic" that suits the current network. In this paper, we study this heuristic learning paradigm for link prediction. First, we develop a novel -decaying heuristic theory. The theory unifies a wide range of heuristics in a single framework, and proves that all these heuristics can be well approximated from local subgraphs. Our results show that local subgraphs reserve rich information related to link existence. Second, based on the -decaying theory, we propose a new method to learn heuristics from local subgraphs using a graph neural network (GNN). Its experimental results show unprecedented performance, working consistently well on a wide range of problems.

Quality:   From the technical point of view, the paper seems to be well-prepared. In particular, the authors propose $\gamma-heuristic theory$ for link prediction and show that many common heuristics can be expressed in a general formulation.  They build their framework based on this theory and on the previous work  (WLNM). However,  I would like to touch on some points here: - In Line 115, there is no clear definition of the \textbf{h-order heuristic} although it is mentioned between lines 26 and 31. If \textbf{h-order heuristic} is defined as, for example, "a heuristic requiring to know up to h-hop neighborhood of the target nodes", then do we really need Theorem 1? - In lines 274 and 275, it is stated that "We run all experiments for 5 times and report the average". It would be better if more runs are performed. - Lines 278-283: The training set size consists of the 90\% of all edges. I think it could be better to see the performance of the framework over different training/test set ratios. (In Appendix, there is another experiment similar to the one in Node2Vec paper in which 50\% of the edges are chosen as training set). - The standard deviation of experiments can also be included in the tables of the paper, as it is done in the Appendix. - The explicit features are not used in Table 2. I think it could be better to perform an experiment including explicit features. - The $\gamma$-heuristic theory is only used to show that small $h$ values can be sufficient to approximate some heuristics. It could be interesting that if the authors could develop a method that is able to learn the parameters within the general formulation.  Clarity:   I think that the structure and readability of the paper is quite good. The authors clearly define the \textbf{h-order heuristic} theory, with several examples. Then, they shortly mention about the methods that have been used in their framework. Here, I would like to indicate some mistakes in the notation: - In Lines 143 and 172, two different walk and length representations $|(w_1,...,w_n)|$ and $len()$ are used. - In Line 143, $d(i,x)$ and $d(i,y)$ should be $d(v_i,x)$ and $d(v_i,y)$ since $i$ is index. - In Line 143, the expression $l=|\{x,v_1,...,v_{i-1}\}| + |\{v_{i+1},...,v_{l-1},y\}|$ should be $l=|\{x,v_1,...,v_{i-1},v_{i}\}| + |\{v_{i}, v_{i+1},...,v_{l-1},y\}|$ - In Line 264, "This" should be changed with "These"  Originality: The authors give a nice general formulation (\gamma-heuristic theory) for link prediction heuristics and show that many heuristics can be approximated with small $h$ values and appropriate parameters. Although it seems that the rest of the paper is similar to the WLNM method, they have managed to enhance it, outperforming previous state-of-the-art methods by changing the methodologies used at the different steps. Unlike WLNM, it also enables the use of explicit and latent features of nodes.  Significance: As it was stated above, one of the major contributions of this work is the development of the $\gamma$-decaying heuristic theory which stems from the natural design of many heuristics. The proposed framework also allows us to benefit from the latent and explicit node features with the favor of GNN model in link prediction tasks and the experiments performed over many datasets also support its significance.


Faster Neural Networks Straight from JPEG
The simple, elegant approach of training convolutional neural networks (CNNs) directly from RGB pixels has enjoyed overwhelming empirical success. But could more performance be squeezed out of networks by using different input representations? In this paper we propose and explore a simple idea: train CNNs directly on the blockwise discrete cosine transform (DCT) coefficients computed and available in the middle of the JPEG codec. Intuitively, when processing JPEG images using CNNs, it seems unnecessary to decompress a blockwise frequency representation to an expanded pixel representation, shuffle it from CPU to GPU, and then process it with a CNN that will learn something similar to a transform back to frequency representation in its first layers. Why not skip both steps and feed the frequency domain into the network directly? In this paper, we modify libjpeg to produce DCT coefficients directly, modify a ResNet-50 network to accommodate the differently sized and strided input, and evaluate performance on ImageNet. We find networks that are both faster and more accurate, as well as networks with about the same accuracy but 1.77x faster than ResNet-50.

My main concerns were responded to in the the rebuttal (separation between IO/CPU/GPU gains), TFLOP measurements and discussion of related works.  I was happy to see that their model (Late-Concat-RFA-Thinner) is still faster than ResNet 50 (approx 680/475= 43% gains in Fig. 1. c) rebuttal instead of 365/205= 78% in Fig. 4. paper) when feeding the models with zeros disregarding JPEG decoding (CPU time) and IO costs. This is a pessimistic estimate given that the ResNet 50 RGB needs to also do the inverse DCT to go from DCT coefficients to RGB domain.  However, I was a bit surprised to see such a big disconnect between the timing numbers and the TFLOP measurements (Fig. 1. b vs Fig 1. c rebuttal). While I trust that the authors timed the models fairly and thus I do not doubt the results, I think this would be worth more investigation.  For the related works, the authors did a good discussion of them in the rebuttal, but I find it strange that we had to ask for this. (This is something that should obviously be in the initial version of the paper!).  Overall however, accounting for everything, this sentence in the rebuttal resonated with me: "We suggest that without our paper, anyone wanting to run modern deep networks on JPEG representations would first have to re-discover or re-engineer all these critical details that comprise the majority of our paper."  Thus I'm keeping my rating to accept the paper.  ================================  The paper presents a detailed study on a simple but old idea: given that most images are stored as JPEGs, and that the JPEG decoder is nothing but a linear transformation over 8x8 blocks, can we not avoid the JPEG decoding step? There are prior works that have explored similar ideas (which should be properly cited and discussed!), like: * On using CNN with DCT based Image Data Matej Ulicny & Rozenn Dahyot, IMVIP 2017 * Using Compression to Speed Up Image Classification in Artificial Neural Networks Dan Fu, Gabriel Guimaraes, 2016 * Towards Image Understanding from Deep Compression Without Decoding, Torfason et al. ICLR 2018 * High speed deep networks based on Discrete Cosine Transformation, Zou et al, ICIP 2014.  However, aside from Torfason et al. (which do not focus on JPEG but deep compression), these papers are limited in the setting (not going far beyond MNIST classificatoin). In contrast, this paper thoroughly studies it in a large scale setting, showing practical gains for ImageNet classification, exploring the architecture space, e.t.c.  However, I still have some concerns about the paper: * Related works needs more discussion * FPS measurements would be well complemented with TFLOP numbers  Another concern I have is the following: are we seeing implementation speedups or computational speedups? The JPEG decoder is decoding on CPU, whereas the the network is on the GPU. By feeding DCT coefficients to the network, you reduce the CPU workload, so are we just seeing speedups from this workload being moved over to the GPU? The proper way to compare with ResNet-50 RGB would be to use the modified libjpeg created by the authors, and then implement the DCT - > RGB decoding step with a fixed 8x8 deconvolution (carefully checking that it gives the same output as standard JPEG decoder). This should speed up the ResNet50 RGB model, no?  Overall the paper is well written and easy to follow and presents speedup on a standard classification pipeline that are significant.


Object-Oriented Dynamics Predictor
Generalization has been one of the major challenges for learning dynamics models in model-based reinforcement learning. However, previous work on actionconditioned dynamics prediction focuses on learning the pixel-level motion and thus does not generalize well to novel environments with different object layouts. In this paper, we present a novel object-oriented framework, called object-oriented dynamics predictor (OODP), which decomposes the environment into objects and predicts the dynamics of objects conditioned on both actions and object-to-object relations. It is an end-to-end neural network and can be trained in an unsupervised manner. To enable the generalization ability of dynamics learning, we design a novel CNN-based relation mechanism that is class-specific (rather than object-specific) and exploits the locality principle. Empirical results show that OODP significantly outperforms previous methods in terms of generalization over novel environments with various object layouts. OODP is able to learn from very few environments and accurately predict dynamics in a large number of unseen environments. In addition, OODP learns semantically and visually interpretable dynamics models.

This paper addresses the problem of action-conditional video prediction via a deep neural network whose architecture specifically aims to represent object positions, relationships, and interactions. The learned models are shown empirically to generalize to novel object configurations and to be robust to minor changes in object appearance.  Technical Quality  As far as I can tell the paper is technically sound. The experiments are well-designed to support the main claims. I especially appreciated the attempts to study whether the network is truly capturing object-based knowledge as a human might expect (rather than simply being a really fancy pixel -> pixel model).  Clarity  Honestly, this paper is a difficult read. A significant chunk of it is a description of a very complicated neural network architecture and I found it very hard to follow. That said, the architecture itself is one of the primary contributions and it must be described. Further, space limitations prevent a more relaxed walkthrough of the approach. Essentially, I'm afraid that I don't have any brilliant ideas about how to improve the reader's experience. I guess I would just say to the authors that it would be valuable for them to go over Section 3 with an eye toward giving high-level intuition and grounding the architectural choices in examples. If the authors can give the reader anything additional to help build this structure in their head and understand the intuitive function of each component, it will help.  Originality/Significance  The problem the paper is attempting to tackle is important and of broad interest. While certainly many of the components of the approach are familiar, the real trick is in composing them in a way that they will work together. The result is what seems to me to be a significant achievement and definitely a novel contribution. This approach can start with pixels, segment the scene into intuitive objects/object classes, and verifiably apply learned and generalizable object relations to predict movement. I'd say that's nothing short of remarkable; people are going to want to hear about this.  I will say, however, that I do wish the paper had spent a little bit of time discussing the weakness/limitations of the approach. What are some types of dynamics that it is not likely to handle well and what are the core challenges? For instance, based on my understanding, it seems that it would not be well-suited to problems where objects can appear and/or disappear (since it only predicts movement for each object already on the screen). It would just be nice to look up and survey what this approach addresses and what significant hurdles remain.  Overall  This paper represents a significant advance in the challenge problem of action-conditional video prediction. Though I wish it were easier to extract high-level intuition and big ideas from the description of the network architecture, I understand the challenge of clearly presenting such a complicated structure. The approach is well-evaluated, experiments validating that the model is able to exploit object-relational structure as intended.  ---After Author Response---  Having considered the other reviews and the author response I have decided to retain my original score. I frankly do not understand the arguments of the other reviewers, who seem to be dismissing the novel and important contribution of the paper (an end-to-end architecture that verifiably develops an object-oriented representation from raw perception) and demanding that the paper solve a bunch of other really hard, open problems (like dealing with variable numbers of objects, large numbers of object types, dynamic backgrounds, agent-centric positions, etc.). I continue to be impressed by the results and satisfied by the well-designed experiments that investigate the capabilities and behavior of the network. Though I frankly don't see where there's room, if the authors can squeeze the Mars Rover experiment into the paper as well, that would alleviate concern that this is a one-trick pony.  To the authors, I suppose I can only say that the kind of discussion that is present in the response about what the model can and cannot do and how one might imagine extending/building upon these ideas to handle more complicated/interesting problems is precisely what the paper needs. If this paper is rejected, which would be a shame in my estimation, I strongly encourage the authors to submit this paper again. Perhaps a robust discussion along these lines in the paper itself would help to highlight the significance of the contribution in the context of these bigger goals.


Learning to Teach with Dynamic Loss Functions
Teaching is critical to human society: it is with teaching that prospective students are educated and human civilization can be inherited and advanced. A good teacher not only provides his/her students with qualified teaching materials (e.g., textbooks), but also sets up appropriate learning objectives (e.g., course projects and exams) considering different situations of a student. When it comes to artificial intelligence, treating machine learning models as students, the loss functions that are optimized act as perfect counterparts of the learning objective set by the teacher. In this work, we explore the possibility of imitating human teaching behaviors by dynamically and automatically outputting appropriate loss functions to train machine learning models. Different from typical learning settings in which the loss function of a machine learning model is predefined and fixed, in our framework, the loss function of a machine learning model (we call it student) is defined by another machine learning model (we call it teacher). The ultimate goal of teacher model is cultivating the student to have better performance measured on development dataset. Towards that end, similar to human teaching, the teacher, a parametric model, dynamically outputs different loss functions that will be used and optimized by its student model at different training stages. We develop an efficient learning method for the teacher model that makes gradient based optimization possible, exempt of the ineffective solutions such as policy optimization. We name our method as "learning to teach with dynamic loss functions" (L2T-DLF for short). Extensive experiments on real world tasks including image classification and neural machine translation demonstrate that our method significantly improves the quality of various student models.

** Post Response ** A thorough discussion regarding the connections to meta-learning techniques [3-7] should be added to the paper (and would significantly strengthen the paper, as it would make it more accessible and more likely to have impact within the meta-learning community). I hope that the authors include this discussion in the final version, as they say they will in the author response.  The new experiments and results address my other primary concerns, demonstrating that the method performs very strongly on a number of challenging, large-scale problems and generalizes to new datasets and architectures. I am raising my score from a 7 to an 8.  ** Original Review ** This paper aims to learn a differentiable loss function that leads to good performance on the "true" objective for a particular task (e.g. accuracy or BLEU score). In particular, the loss function is optimized with gradient descent with reverse mode autodiff w.r.t. the true objective on held-out data. The experiments include comparisons to prior work on MNIST, CIFAR-10, CIFAR-100, and De-En NMT, showing a small improvement.  Pros: The idea is interesting and motivated well, from the perspective of a teacher adapting how it teaches a student. The experiments include a number of different datasets, many different model architectures, and comparisons to multiple previous methods. Cons: The improvement over prior work seems to be relatively small (but it is consistent across tasks). There are important aspects of the experimental set-up that are unclear (details below). Finally, the proposed approach seems to be very computationally intensive, since it requires training the teacher on the task and then training the student using the teacher. After being trained, can the teacher generalize to new tasks?  Method: As mentioned above, the method is nicely motivated, from the perspective of a teacher. But, the method in its current form seems somewhat impractical, as it requires training a teacher from scratch for a particular task and then training the student. It seems like it would be much more practical to train the teacher across multiple tasks and measure generalization to new tasks. Can the teacher generalize to new settings? Regarding reproducibility, will code be released?  Experiments: How does the proposed approach compare to reward-augmented max likelihood [1] and distillation [2]? [1] considered the same problem of optimizing for a non-differentiable reward function. Model distillation [2] also has a similar flavor of a teacher/student where the teacher is an ensemble of models and the student is a single model. One crucial piece of information that is missing from the experiments is -- what data is used for the development set, and how was this data used for the comparisons? The concern is that, perhaps the proposed method is seeing a benefit because it is using the validation set in a more direct way, rather than simply being used to find good hyperparameters. If the dev set is only used by the prior methods for choosing hyperparameters, how do these prior methods perform when training with both the training and development data (after finding good hyperparameters by holding out the dev set)? Additionally, when experimenting with multiple student architectures, was a single teacher trained across all student architectures? Or was it trained individually for each architecture? Or was the teacher trained on one student architecture and evaluated on another? This bit of information doesn't seem to be stated in the paper. Showing generalization to new student architectures would be nice.  Related Work: It would be nice (but not crucial) to discuss the relationship with works on learning to optimize [3,4]. These works are related in that they learn a meta-learner/teacher that learns how to update learner/student. Unlike this work, that learns a loss function, these prior works learn to directly output the weight update to be applied to learner/student. There is also prior work in learning-to-learn that learns critics or loss functions across multiple tasks [5,6,7]. Of course, one big difference with this work and these prior works is that these prior works typically train over multiple tasks or multiple architectures, evaluating generalization to new settings. This paper also seems to be related to automatic curriculum generation, e.g. [8].  Minor comments: - The notation for y in Section 4.1 is inconsistent. What is the difference between y, y with an arrow, and y' with an arrow?  [1] Reward Augmented Maximum Likelihood for Neural Structured Prediction https://arxiv.org/abs/1609.00150 [2] Distilling the Knowledge of a Neural Network https://arxiv.org/abs/1503.02531 [3] Learning to Learn by Gradient Descent by Gradient Descent https://arxiv.org/abs/1606.04474 [4] Learning to Optimize https://arxiv.org/abs/1606.01885 [5] Meta-critic Networks https://arxiv.org/abs/1706.09529 [6] Domain Adaptive Meta-Learning https://arxiv.org/abs/1802.01557 [7] Evolved Policy Gradients https://arxiv.org/abs/1802.04821 [8] Automatic Curricula via Asymmetric Self-Play https://arxiv.org/abs/1703.05407


Deep Defense: Training DNNs with Improved Adversarial Robustness
Despite the efficacy on a variety of computer vision tasks, deep neural networks (DNNs) are vulnerable to adversarial attacks, limiting their applications in security-critical systems. Recent works have shown the possibility of generating imperceptibly perturbed image inputs (a.k.a., adversarial examples) to fool well-trained DNN classifiers into making arbitrary predictions. To address this problem, we propose a training recipe named "deep defense". Our core idea is to integrate an adversarial perturbation-based regularizer into the classification objective, such that the obtained models learn to resist potential attacks, directly and precisely. The whole optimization problem is solved just like training a recursive network. Experimental results demonstrate that our method outperforms training with adversarial/Parseval regularizations by large margins on various datasets (including MNIST, CIFAR-10 and ImageNet) and different DNN architectures. Code and models for reproducing our results are available at https://github.com/ZiangYan/deepdefense.pytorch.

The main idea of this paper is contribution of training recipe named "deep defense" that uses modification of classification loss with perturbation based regularizer and uses the encoder/decoder based architecture to achieve results on MNIST, CIFAR-10 and Image Net datasets that surpass ones from chosen reference implementation, DeepFool and FGS adversarial implementations and shows robustness that surpasses ones from above adversarial algorithms as well, by a wide margin.  The paper builds on the perturbation based regularization method for adversarial training with non-equal (weighted) treatment of robust examples (versus the others). The algorithm is described well, though sometimes than referring to the reference for the important details, would be good to include them, esp. in context of the setup, formulae and notations. The authors do a good job of supporting their reasoning for the choice of the perturbation regularization term, that has the advantage of utilizing the network architecture without any changes, and can be utilized in encoder/decoder framework, with initializations from pre-trained networks to achieve robustness to adversarial attacks while maintaining high accuracy on benign examples. The claims from the authors are well supported  by the results in Table 1 and 2 and the Figures 2-4 that show that Deep Defense beats the other two adversarial methods (Deepfool and FGS) methods on MNIST, CIFAR-10 datasets and is more robust across the board and also surpasses the reference network itself in benign accuracy including the ImageNet. They also demonstrate the impact of changing the hyperparameters c and lambda and layer regularization and how the accuracy varies with epochs. Authors intend to open source implementation.  A few areas of explanations and improvements that would help understanding the paper better are mentioned below  - It would be really interesting to add if the authors faced any challenges with deconvolution. In the literature both upsampling/convolution and deconvolution/unpooling have been used in decoders and both have pros/cons, though in the encoder/decoder setup (which the authors for some reason don't call as such), deconvolutions were employed by the authors and would be great to know if there were any challenges (esp. with deeper networks). Also, conspicuously the details on the decoder is missing from the description in the supplement. - details on R the exponential function in the perturbation regularization term aren't provided (its just mentioned as such) - the paper mentions all operations in CNN being differentiable and mention ReLU, that sounds off-point  - What is the stopping criteria (different epochs/batches/iterations are mentioned for different datasets but it wasn't clear  - In Table 1 the Perturbed examples from DeepFool were chosen when the Deep Defense drops to 50%, would good to know a) what if the perturbed data was instead chosen from one of the other two adversarial algorithms (so for e.g. when their performance drops to 50%) b) what happens below 50% was chosen to start with - For Image Net the authors conclude insufficient hyper-parameter exploration to include results for the two adversarial algorithms, would be good to include that in final version - The choice of the networks, why Alexnet and Resnet-18 were chosen (the authors make a remark about deep neworks, and would like to get their input on deeper architectures)


Learning to Multitask
Multitask learning has shown promising performance in many applications and many multitask models have been proposed. In order to identify an effective multitask model for a given multitask problem, we propose a learning framework called Learning to MultiTask (L2MT). To achieve the goal, L2MT exploits historical multitask experience which is organized as a training set consisting of several tuples, each of which contains a multitask problem with multiple tasks, a multitask model, and the relative test error. Based on such training set, L2MT first uses a proposed layerwise graph neural network to learn task embeddings for all the tasks in a multitask problem and then learns an estimation function to estimate the relative test error based on task embeddings and the representation of the multitask model based on a unified formulation. Given a new multitask problem, the estimation function is used to identify a suitable multitask model. Experiments on benchmark datasets show the effectiveness of the proposed L2MT framework.

The authors of the manuscript consider the problem of selecting an information transfer algorithm in the multi-task setting. Instead of performing some kind of model selection, they propose a learning-based approach: based on historical MTL experience they build a mapping from MT training data and MT model into relative test performance improvement that using that MT model brings compared to solving each task in isolation. For a new, test, MTL problem they optimise the resulted function with respect to MT model to maximise the improvement.  Pros: This is a natural continuation of multi-task/learning to learn ideas with adding one more layer in the hierarchy  that has been mentioned in works of Baxter [7]. The authors' implementation of it relies on a nice unification of various MTL methods presented in section 2.  Cons: The practicality of the proposed L2MT seems rather limited to me - MT experiences used for training need to be very similar to the problems appearing at the testing stage. Also, data-embedding LGNN seem to require all experiences to be MTL problems with the same number of dimension. Consequently, in the experimental evaluation both training and test MTL problems come from the same dataset. This, in my opinion, greatly limits applicability of L2MT in realistic scenarios, especially since the number of training experiences has to be significant.  It would be interesting to see what happens if test and training problems are coming from different datasets, for example, by merging MIT-Indoor-Scene and Caltech256 experiments.  Experiments: - how parameters were selected for the baselines? - how MT models and relative test errors were obtained for the training stage of L2MT?  Additional comments: - I believe, in Theorem 4 one needs the MTL experiences to be sampled from a distribution, not individual tasks. Also, without a more direct bound on the Gaussian width of {E_i} the resulting bound is not particularly conclusive - the authors may want to add [1] to the related work, as it is based on a similar idea but for domain adaptation  [1] Transfer Learning via Learning to Transfer, Ying WEI, Yu Zhang, Junzhou Huang, Qiang Yang ICML'18  Update: based on the authors' response and other reviews I raise my score to 6


Learning filter widths of spectral decompositions with wavelets
Time series classification using deep neural networks, such as convolutional neural networks (CNN), operate on the spectral decomposition of the time series computed using a preprocessing step. This step can include a large number of hyperparameters, such as window length, filter widths, and filter shapes, each with a range of possible values that must be chosen using time and data intensive cross-validation procedures. We propose the wavelet deconvolution (WD) layer as an efficient alternative to this preprocessing step that eliminates a significant number of hyperparameters. The WD layer uses wavelet functions with adjustable scale parameters to learn the spectral decomposition directly from the signal. Using backpropagation, we show the scale parameters can be optimized with gradient descent. Furthermore, the WD layer adds interpretability to the learned time series classifier by exploiting the properties of the wavelet transform. In our experiments, we show that the WD layer can automatically extract the frequency content used to generate a dataset. The WD layer combined with a CNN applied to the phone recognition task on the TIMIT database achieves a phone error rate of 18.1%, a relative improvement of 4% over the baseline CNN. Experiments on a dataset where engineered features are not available showed WD+CNN is the best performing method. Our results show that the WD layer can improve neural network based time series classifiers both in accuracy and interpretability by learning directly from the input signal.

* Summary:  This paper introduces a wavelet deconvolution (WD) layer for networks working on time-series. The WD performs a linear convolution with parametrized filters, using a closed-form formula which only involves one parameter per filter, the scale. The scale parameter varies continuously and can be adapted by gradient descent, removing the need for a preprocessing with e.g. mel filterbanks. A synthetic experiment on a toy classification example with time-frequency content shows that this layer improves results with respect to a CNN working directly on the raw waveform. Classification results on speech (TIMIT) show that accuracy is improved for a similar CNN architecture when the hand-crafted preprocessing is replaced by the WD layer, yielding results not too far from the state-of-the-art. Classification results on a very small dataset (UCR Haptics, 10x smaller than TIMIT) yield state-of-the-art classification results.  * General opinion about the paper:  This paper is quite well written, up to minor remarks. The general idea is clear.  This paper is in the same vein as other recent papers which try to bridge the gap between hand-crafted features and completely black-box methods on raw data (see e.g. [Learning filterbanks from Raw speech for Phone Recognition, Zeghidour et al, ICASSP 2018]), except that it uses closed-form filters with few parameters, which allows to improve the regularity of the learnt filterbank.  The authors could have more insisted on the potential impact of this method for dealing with very small data. The experiments support the claims of the authors, even if a deeper analysis in some parts would have been appreciated. The experiments are almost reproducible by reading the paper (cf remarks below).  The main criticism to me lies in the lack of generality of the paper, which limits its significance. As mentioned in the conclusion by the authors, the method would be applicable to many different kinds of parametrized time-frequency transforms, as long as the parameters vary continuously (cf remarks below): I would have appreciated such a general formulation. The Mexican hat wavelets could still have been used as an illustration. Also, the name "wavelet deconvolution" appears very ill-chosen to me, insofar as the layer precisely performs a convolution with a wavelet filterbank; this has nothing to do with the deconvolution problem $y = A \ast x$ where x is unknown.  * Detailed remarks:  1) Line 63, $F_s = F_c / (s \Delta)$. What is $\Delta$? It is not introduced.  2) It would be nice to give in 1 sentence what the straightforward extension to the multi-channel signal (l. 95-96) would be, because it is not obvious to me. Does it consist in applying this transform to each channel separately?  3) If I understood correctly, the WD layer performs a linear operation. Spectrograms are typically computed after a non-linear transform z_i = |x \star h_i| where the h_i are the different filters of a local Fourier family (possibly on the mel scale). Despite this difference, the two operations are very often compared in the paper (cf Figure 1 for instance). Why not including the non-linearity here?  4) What motivates the choice of the Mexican Hat wavelets? Indeed, as it is real and symmetric, it cannot be analytical in Fourier: this makes it very different from the standard mel filterbanks used in audio.  5) I think that the arguments supporting the use of the wavelet transform (l.257-262) could be applied to other transforms, as long as they have continuous parameters. It is true that the wavelet parametrization in time-frequency is already good for many applications, but allowing more flexibility with more parameters would be useful in many cases.  6) The derivation of the gradient l.106-111 is not essential to the paper, especially with the use of automatic differentiation software. I would recommend to leave it in the appendix in order to leave more space for the rest of the paper.  7) In Figure 2, right panel, the scale parameters seem to be already very close to the value which is chosen. How is this possible with a random initialization?  8) The nature of the initialization used (l.118) would require a clarification. Are the frequencies of the wavelets F_s evenly spaced (and the scales s adapted to match this spacing)? Or are the scales s themselves evenly spaced?  9) In the TIMIT experiment, why is it necessary to add the $\delta$ and $\delta\delta$ values of the WD to the network? What was the performance without it? How many filters are used and how does it compare to the number of mel filterbanks features used in the other methods? This hyperparameter value is missing (or is there a typo in the caption of Figure 2: "of (the) 10 filters"?). An ablation experiment would also be interesting for this dataset: what would be the performance of the network with a random initialization of the filters, without tuning their scales?  10) Figure 2 is not that illuminating. Since the use of this layer is the alleged interpretability of the layer, it would have been nice to simply show the set of all scales (possibly sorted). Do they follow a geometric progression? Or an arithmetic one? Further, the comment related to the shape of the filters in the caption is slightly misleading: the shape of the filters is defined by the choce of the Mexican hat wavelet (which is then dilated), it has nothing to do with the training.  11) In the UCR experiment, how does the proposed method compare to the others in terms of numbers of parameters and training time?  12) The first paragraph of the conclusion should be moved somewhere else.  13) I think that there is a minor typo in the Equation after line 98: x_{j+k} should be replaced with x_{j-k}.


Removing the Feature Correlation Effect of Multiplicative Noise
Multiplicative noise, including dropout, is widely used to regularize deep neural networks (DNNs), and is shown to be effective in a wide range of architectures and tasks. From an information perspective, we consider injecting multiplicative noise into a DNN as training the network to solve the task with noisy information pathways, which leads to the observation that multiplicative noise tends to increase the correlation between features, so as to increase the signal-to-noise ratio of information pathways. However, high feature correlation is undesirable, as it increases redundancy in representations. In this work, we propose non-correlating multiplicative noise (NCMN), which exploits batch normalization to remove the correlation effect in a simple yet effective way. We show that NCMN significantly improves the performance of standard multiplicative noise on image classification tasks, providing a better alternative to dropout for batch-normalized networks. Additionally, we present a unified view of NCMN and shake-shake regularization, which explains the performance gain of the latter.

Summary: This paper presents an analysis of multiplicative noise based regularizers (e.g. DropOut) and demonstrates that employing multiplicative noise leads to internal features becoming correlated, as the regularized objective favors increasing the signal-to-noise ratio by adding redundancy. The authors propose to decompose the pre-activations of the noised layers into their signal and noise components (which is tractable since the noise under consideration is artificial) and remove the effects of the noise from the backward pass. They show that this removal changes the regularized objective to not encourage feature correlation (although it does not, as far as I can tell, penalize it, either). A connection is drawn between the proposed regularizer and shake-shake (another backward-pass-hacking multiplicative-noise based regularizer), providing (for the first time I’m aware of) some analytical justification for the latter. The results are justified through standard benchmarks on CIFAR, where they demonstrate competitive (potentially even SOTA) results for their method.  The paper is well-motivated, well-written, clear, and easy to follow. The technique proposed is simple and should be easy to implement and reproduce, without requiring more than small architectural changes. I found the analytical results convincing, and the connection to shake-shake (which, along with its variants, has thus far been an ad hoc hack) worthwhile. I found the empirical results as acceptable (the scores on CIFAR are certainly good) but lacking error bars on test error (which I don’t believe we should be letting authors get away with) or results on larger-scale tasks, particularly ImageNet (see my comments below). I think this is worth the attention of the NIPS community and vote in favor of acceptance.  General notes:  -The insight that features can be decomposed into the noise and signal (because the noise is artificial) is intriguing and clever. I found equation 15 particularly elegant.  -Please re-run your CIFAR experiments and report error bars. While common practice may be to *not* do this, I don’t think reviewers should be letting authors get away with this in situations where it should obviously be done. Even on a modest GPU these experiments can be run in under a day, so it should be no challenge to at least take the average of 3 and report variances for at least the top-scoring run.  -My primary complaint with this paper and with the shake-shake line of research is that no one, to my knowledge, has demonstrated that these techniques work at a scale beyond CIFAR. CIFAR-10 in particular has 4 times as many images per class as ImageNet, and it's known to be a regularization game, but for other tasks which are perhaps more practical, it’s not clear that these techniques provide any benefit. I don’t *fault* the authors for not running ImageNet experiments as it would be unreasonable to expect every group to test on large scale tasks, but I suspect that if the authors want to convince the broader community of the value of their work, they could easily do so by being the first to demonstrate that these techniques help on ImageNet (or that they don’t—such a negative result would also improve this paper).    -Calling it Feature-Decorrelating multiplicative noise may not be quite correct—as far as I can tell (and the authors are welcome to correct me on this front), nothing in the regularized objective actually encourages decorrelation between features. A more accurate name might be something along the lines of "Non-Correlating Multiplicative Noise."  While it appears that in practice variants 0 and 1 have lower feature correlations than the un-noised baseline, variant 2 often has slightly higher feature correlations (though lower than the MN baseline). This is not discussed anywhere in the paper that I can find, and I think it should be. I’m curious as to the authors’ opinion on this anyhow—is it because the noise is only applied once every two layers, or what?   -What exactly are the error bars in the bar graphs of e.g. figure 1? Are they representing the variance of feature correlation across runs with multiple different settings, or across different layers at a given resolution in a single run? Please state this explicitly in the paper.  -I found the description of hyperparameters confusing. The statements “the learning rate is set to either 0.03 or 0.04” implies that either different settings are used for different experiments, or grid search was used and the results were then averaged, but this is not clarified anywhere. If the authors wish to move hyperparameter details into supplementary material to save space in the body of the paper, that’s fine, but the way it is currently written is vague and might hinder reproduction.   -The authors might consider leveraging empirical successes of MN strategies as an auxiliary justification for using batchnorm—currently batchnorm is often used because “it just works,” but if one finds that FDMN is useful but only works when batchnorm is in the mix, then one might justify batchnorm as “I need FDMN and it only works with batchnorm, a combination which is justified by the analysis in the FDMN paper.”  -I briefly tried coding up FDMN-0 several different ways for WRN-40-4 and WRN-28-10 but didn’t really see a change in performance on CIFAR-100+. I didn't try very hard, so there could of course be mistakes in my implementation, and I have not factored this into my score . One implementation detail that was not clear to me was how to apply this to pre-activation style ResNets—should the noise be added at the batchnorm’d outputs before nonlinearities, without any other changes, or should the batchnorm orders be changed (an additional BN is mentioned for variants 1 and 2)?  -My implementation also made me wonder if the authors truly mean that they set the standard deviation of their uniform noise to 0.25 for CIFAR-100—this results in uniform noise U(-sqrt(3)/2, sqrt(3)/2),  which seems like a very large noise component.   -Minor:   -line 19, “However, such over-parameterized models are more prone to overfitting.” It’s not presently clear that this is actually true; the existence of a generalization gap does not necessarily imply overfitting, and models often continue to improve their validation score even as they squeeze the training loss ever closer to zero.  -line 132: I found the use of parentheses in “mean (variance)” momentarily confusing. I take it to mean “mean and variance.”  -The metric chosen for feature correlation seems reasonable, though in an ideal world it would be good to try methods particularly targeted at analyzing the similarity of internal representations in deep nets, e.g. SVCCA [NIPS 17]. The authors needn’t feel obliged to do this, though.    Response to author feedback: I feel that the authors' response has adequately addressed my concerns and my inclination is to score this paper as more of an 8 than a 7. I think that the line of work on "gradient-hacking" regularizers like Shake-Shake and ShakeDrop is interesting and potentially very valuable (the techniques show a lot of promise). Recent papers in this line of research have been rejected from the mainstream conferences, perhaps because the papers themselves were somewhat lacking. This method introduces fewer design choices (the choice of distribution and the variance of the noise) than the Shake* regularizers, which have myriad moving parts and necessitate architectural changes. As a practitioner I would much rather drop this noise into my code. To me this paper does a good job of justifying this style of technique by properly connecting it to better-understood existing methods, provides mathematical detail but doesn't fall prey to "mathiness", has good, although not quite "perfect," experimental results, and (apparently) has open source code. I argue more strongly in favor of acceptance.


Random Feature Stein Discrepancies
Computable Stein discrepancies have been deployed for a variety of applications, ranging from sampler selection in posterior inference to approximate Bayesian inference to goodness-of-fit testing. Existing convergence-determining Stein discrepancies admit strong theoretical guarantees but suffer from a computational cost that grows quadratically in the sample size. While linear-time Stein discrepancies have been proposed for goodness-of-fit testing, they exhibit avoidable degradations in testing power-even when power is explicitly optimized. To address these shortcomings, we introduce feature Stein discrepancies ( SDs), a new family of quality measures that can be cheaply approximated using importance sampling. We show how to construct SDs that provably determine the convergence of a sample to its target and develop high-accuracy approximations-random SDs (R SDs)-which are computable in near-linear time. In our experiments with sampler selection for approximate posterior inference and goodness-of-fit testing, R SDs perform as well or better than quadratic-time KSDs while being orders of magnitude faster to compute.

The authors have addressed most of my concerns, and I maintain my overall evaluation.  ----------  Based on recent work in graph Stein discrepancy and kernel Stein discrepancy (KSD), this paper proposes feature Stein discrepancies (\PhiSDs) as a new family of discrepancy measures that can be efficiently approximated using importance sampling (termed R\PhiSDs) which can be computed in near-linear time. By finding suitable KSDs that are upper-bounded by \PhiSDs, the paper shows that \PhiSDs provably determine the convergence of a sample to its target. The application of R\PhiSD to goodness-of-fit testing is discussed, and its asymptotic distribution is derived. Experiments are conducted on sampler selection in approximate inference as well as goodness-of-fit testing.  Overall, the writing of the paper is fairly clear and the mathematical development quite rigorous, although at times it would help to provide some more intuition/motivation for the definitions/assumptions being introduced (such as the name “tilted base kernel” and the proposed hyperbolic secant kernel).  I believe that this work is a nice addition to the existing literature on Stein discrepancies and goodness-of-fit testing. In particular, it is interesting to see that \PhiSDs include KSD and the finite-set Stein discrepancy (FSSD) as special cases.  More specifically, I have the following specific questions/comments:  - Since it is not a commonly used kernel function, what is the intuition and motivation for considering the hyperbolic secant kernel? Is it merely for mathematical convenience for verifying the various assumptions or are there certain application scenarios in which it would arise as a natural choice?  - If one wishes to develop another instance of R\PhiSD, rather than checking all of Assumptions 3.1 through 3.6, is there a more concise but possibly slightly stronger set of conditions that could be more intuitive or easier to verify in practice and guide the choice of the various quantities in the definition of R\PhiSD?  - While the authors have given two examples of R\PhiSDs, is it possible to provide a general characterization of how the choice of r affects the properties of the resulting R\PhiSD?  Regarding the experiments:  - In Figure 3(b) and (c), the behavior of L1-IMQ is quite strange, do the authors have any intuition/conjecture behind its counterintuitive behavior as dimension increases? Is this behavior consistent as one increase the number of features used?  - For the computational complexity experiment, while the KSD test serves as a simple sanity check, I would be more interested in seeing how R\PhiSD compares with FSSD, since both should run in near-linear time, and it’s unclear which would be more efficient in practice.  - For the goodness-of-fit testing experiments, it would be interesting and (more informative) to perform experiments on more complicated distributions (in addition to the standard Gaussian), such as those with multiple modes (e.g., Gaussian mixture models). In particular, should one expect better performance of \RPhiSD over KSD and KSSD in the presence of multi-modality?


Maximizing Induced Cardinality Under a Determinantal Point Process
Determinantal point processes (DPPs) are well-suited to recommender systems where the goal is to generate collections of diverse, high-quality items. In the existing literature this is usually formulated as finding the mode of the DPP (the so-called MAP set). However, the MAP objective inherently assumes that the DPP models "optimal" recommendation sets, and yet obtaining such a DPP is nontrivial when there is no ready source of example optimal sets. In this paper we advocate an alternative framework for applying DPPs to recommender systems. Our approach assumes that the DPP simply models user engagements with recommended items, which is more consistent with how DPPs for recommender systems are typically trained. With this assumption, we are able to formulate a metric that measures the expected number of items that a user will engage with. We formalize this optimization of this metric as the Maximum Induced Cardinality (MIC) problem. Although the MIC objective is not submodular, we show that it can be approximated by a submodular function, and that empirically it is well-optimized by a greedy algorithm.

The authors have addressed most of my concerns in their response, and I hope to see the changes they mentioned reflected in the final version.  ----------  The paper proposes an alternative objective for diverse subset selection using determinant point processes (DPPs). For the task of selecting a best subset from the ground set, the authors argue that instead of the maximum a posterior (MAP) objective, a more suitable objective is to maximize the expected number of items in the subset that the user will engage with, which is termed the maximum induced cardinality (MIC) objective. The paper proceeds to investigate theoretical properties of the MIC objective, and proposes an approximation algorithm for optimizing MIC by observing that the objective, while not sub-modular, is fractionally sub-additive.  Overall, the paper is clearly written and appears technically sound. However, while I tend to agree with the intuition that the MIC might be a more suitable objective than MAP in practice, my major concern is that this hypothesis has not been verified/justified with any empirical evidence in the experiments. I believe that this is an important aspect as the hypothesis is not a widely acknowledged fact, and its plausibility greatly affects the significance of the theoretical and methodological contributions made in this paper; thus at least some form of evidence should be provided to justify the motivation of the paper. I understand that as the authors mentioned on line 228, a truly sound comparison between MIC and MAP should be done via a live system on real data, but I believe that simpler simulations or experiments could be enough to shed some light and provide some empirical support for the hypothesis and convince readers of its potential practical importance. Such experiments could be performed on simulated or offline data (e.g., website recommendations/transactions history), or using crowdsourcing tools (e.g., Amazon Mechanical Turk). Some of the evaluation methods used in the DPP papers (Kulesza et al., Gillenwater et al., among others) could also be borrowed here.  On the technical side, I have some more specific comments:  - The authors noted on line 115 that when the DPP kernel is low-rank, MAP can fail dramatically. However, by Corollary 4.1, it seems that when L is low-rank, some eigenvalues of B could be large, which would result in a small value of c, indicating poor approximation quality of \hat{f} to f?  - Line 169: in the summand, what is the index j for?  - In terms of running time complexity, how does the approximate MIC algorithms (GIC, PIC, and SIC) compare with the state-of-the-art approximate MAP algorithm? If SIC is significantly more efficient than the current MAP algorithm, this could also be a potential reason to prefer MIC over MAP.  - Since PIC was used in the experiments, I feel that at least some details regarding the projection onto the space of PSD M-matrices should be included in the main text. In Section 5 of the supplementary material, the authors states that they found “flipping the signs of any positive off-diagonal elements, then projecting to the PSD cone by truncating negative eigenvalues at zero worked best”, and “if the PSD projection resulted in any positive off-diagonal elements, we simply iterated the process of flipping their signs and projecting to the PSD cone until the resulting matrix satisfied all requirements”. I’m not sure if the sign-flipping is a standard/theoretically justified approach of projecting on to the space of M-matrices, and whether the latter iterative procedure would be guaranteed to terminate?  - Minor point: In the experiments using the graph Laplacian, I feel that instead of/in addition to a simple Erdos-Renyi model, using a more realistic random graph model (e.g., the stochastic blockmodel, the preferential-attachment model, etc.) that obeys the many properties found in real-world networks (power-law degree distributions, clustering, small-world property, etc.) could be more informative.