Turbo Learning for CaptionBot and DrawingBot
We study in this paper the problems of both image captioning and text-to-image generation, and present a novel turbo learning approach to jointly training an image-to-text generator (a.k.a. CaptionBot) and a text-to-image generator (a.k.a. DrawingBot). The key idea behind the joint training is that image-to-text generation and text-to-image generation as dual problems can form a closed loop to provide informative feedback to each other. Based on such feedback, we introduce a new loss metric by comparing the original input with the output produced by the closed loop. In addition to the old loss metrics used in CaptionBot and DrawingBot, this extra loss metric makes the jointly trained CaptionBot and DrawingBot better than the separately trained CaptionBot and DrawingBot. Furthermore, the turbolearning approach enables semi-supervised learning since the closed loop can provide pseudo-labels for unlabeled samples. Experimental results on the COCO dataset demonstrate that the proposed turbo learning can significantly improve the performance of both CaptionBot and DrawingBot by a large margin.

Summary: This paper proposed a joint aproach for learning two network: a capitonbot that generates a caption given an image and a drawingbot that generates an image given a caption. For both caption and image generators, the authors use existing network architecture. LSTM - based network that incorporates an image feature produced by Resnet is used for caption generation (the specific architecture is not clearly described). Attention GAN is used to generate an image from caption. The main contribution of this paper is joint training of caption and image generators by constructing two auto-encoders. An image auto-encoder consists of a caption generator feeding an image generator. A caption auto-encoder consists of the same image generator feeding the same caption generator. Both auto-encoders are trained jointly to minimize reconstruction loss of images and captions (in alternative manner).  So for example, in the caption auto-encoder, caption generator is trained to generate captions that not only match the ground truth but are also predictive of the underlying concepts in the image. Same holds for training an image generator.  This joint learning approach is shown to perform better than independent training of caption and image generators on the COCO dataset. Additionally, authors propose a semi-supervised training modification where a small number of captioned images are used with a large number of uncaptioned ones. The proposed joint (turbo) learning approach results in a caption generator that outperforms an independetly trained one since the latter cannot incorporate unlabeled images.  Quality: The idea is interesting and the experimental results validate the advantage of the proposed turbo learning approach. Clarity: The paper is mostly clearly written. However, I did find several places a bit confusing. - When the LSTM-based caption generator is introduced its not clear how the image feature is incorporated in the network. - When the loss for the alternating optimization is explained, I expected it to be a function of the ground truth caption. I guess it’s implicitly buried inside other terms.  But it would be good to be more rigorous in your definitions. - Reconstruction loss between a true image and a generated image is measured via KL divergences w.r.t to distribution of object classes found in an image by a pre-trained image classifier. I think this is an important concepts that deserved more emphasis.  Originality: While previously explored in Machine Translation, the turbo-learning approach of joint training caption and image generation is novel and interesting. Also similiar  “butterfly” architectures have been used in image transfer learning (e.g.  Unsupervised Image-to-Image Translation Networks , Lie et al, 2018) Significance: This idea of jointly training two inverse networks can potentially be applied to other domains where such set up is possible.


Learning to Specialize with Knowledge Distillation for Visual Question Answering
Visual Question Answering (VQA) is a notoriously challenging problem because it involves various heterogeneous tasks defined by questions within a unified framework. Learning specialized models for individual types of tasks is intuitively attracting but surprisingly difficult; it is not straightforward to outperform naïve independent ensemble approach. We present a principled algorithm to learn specialized models with knowledge distillation under a multiple choice learning (MCL) framework, where training examples are assigned dynamically to a subset of models for updating network parameters. The assigned and non-assigned models are learned to predict ground-truth answers and imitate their own base models before specialization, respectively. Our approach alleviates the limitation of data deficiency in existing MCL frameworks, and allows each model to learn its own specialized expertise without forgetting general knowledge. The proposed framework is model-agnostic and applicable to any tasks other than VQA, e.g., image classification with a large number of labels but few per-class examples, which is known to be difficult under existing MCL schemes. Our experimental results indeed demonstrate that our method outperforms other baselines for VQA and image classification.

# Response to authors’ response # I appreciate the authors' effort to include (1) VQA2 (2) an MLP baseline (3) the result at k = 4. Overall, I like the paper’s idea and am satisfied with the response, although the improvement from the independent random ensemble (IE) to the proposed method (MCL-KD) becomes less significant at (1) and (2). The performance of a single model on VQA2 seems to be lower than that reported in [14]. The authors should check on this.  Reviewer 2 and I have several similar concerns---the paper needs to work on real datasets and more algorithms to show its robustness. I strongly suggest the authors to include one more dataset and at least one more algorithm. Also, I would like to see analysis on models learned for VQA2---is each model specialized to a different question type? The paper could bring fruitful thinking and impact to the Visual QA community. I will maintain my score for paper acceptance.  My specific response is in [] after each comment. =========================================================================   # Summary # This paper aims to learn an ensemble of specialized models for Visual QA---each model ideally will be dedicated to different question (or image) styles. The authors propose an objective with distillation on top of the “multiple choice learning MCL framework” to prevent (1) data deficiency on learning each specialized model and (2) forgetting general knowledge. The contribution is complementary and compatible to different Visual QA models (most of the Visual QA papers focus on developing new models), and thus could potentially benefit the whole community.   # Strengths # S1. As mentioned above, the proposed objective is model-agnostic---any existing Visual QA model can potentially be applied to be the specialized model. This is particularly important in the literature of Visual QA---there have been too many papers (at least 30) on proposing different models but the performance gap is very limited and it is hard to tell what are the essential insights / concepts / components for Visual QA. The authors should emphasize this.  S2. The paper is clearly written. The idea is clean and inspiring. The performance improvement compared to the existing MCL and CMCL objectives is significant and consistent.  # Main weaknesses (comments) # W1. The paper only evaluates on a synthetic dataset (i.e., CLEVR). It will be important to evaluate on real datasets like VQA2 or Visual Genome to demonstrate the applicability. It will be interesting to see if each specialized model will be dedicated to different questions types (e.g., questions begin with why, what, how). [The authors do provide results on VQA2, but the improvement is largely reduced. Moreover, the single model performance on VQA2 is lower than reported in [14]. What is the “k” here? In the future, the authors can use the datasets by K. Kafle and C. Kanan, An Analysis of Visual Question Answering Algorithms, ICCV 2017, which has less bias issues.]  W2. To support the strength, it will be important to evaluate on different Visual QA models beyond stacked attention network. A simple MLP baseline is a good starting point. See the following paper as an example to show its applicability. [The authors add the experiments. Although on MLP the gain becomes smaller, it still shows improvement. I would like to see more on this in the final version. Again, what is the “k” here?]  Hexiang Hu, Wei-Lun Chao, and Fei Sha, Learning answer embeddings for visual question answering, CVPR 2018  It will be great if the authors can also discuss other papers that can potentially benefit the whole community such as   Kushal Kafle and Christopher Kanan, Answer-Type Prediction for Visual Question Answering, CVPR 2016  W3. Besides the MCL framework, the authors should discuss other ways to learn an ensemble of models (e.g., mixture of experts) and the possibility (or difficulties) to apply them. [The authors promise to add the discussions.]  # Minor weaknesses (comments) # W4. Eq. (6) seems not mention the case where k > 1. Should it be argmin for top k? Eq. (4) and (5) might be simplified by removing Z and \nu---\nu(Z(x)) is actually P(y|x).  W5. From Table 1, it seems that the larger the k is, the higher the performance is. It would be important to evaluate on k = 4 to see if the performance will still increase or decrease to demonstrate the need of specialized models. [The authors show that the performance degrades when k= 4. Thanks.]  W6. What is M for the image classification experiment?  # Further comments # how to bridge the gap between the oracle and top 1 performance seems to be a promising direction for future work. I’m thinking if changing the decision rule in Eq. (8) by learning an attention p(m|x) on how confidence each model can help. Specifically, if p(m|x) can give higher weight to the model that gives the correct prediction, we can achieve the oracle performance.


Streaming Kernel PCA withÕ( √ n) Random Features
We study the statistical and computational aspects of kernel principal component analysis using random Fourier features and show that under mild assumptions, O( √ n log (n)) features suffice to achieve O(1/ 2 ) sample complexity. Furthermore, we give a memory efficient streaming algorithm based on classical Oja's algorithm that achieves this rate.

=== SUMMARY === The paper deals with efficient computation of kernel PCA using random Fourier features. It proves that under a certain distributional assumption (of exponential spectrum decay) on the input data, ~O(sqrt(n)) random features suffice for a KPCA algorithm to converge to optimum. The framework is instantiated with two algorithms: ERM and Oja's algorithm.  === COMMENTS === The topic of the paper is significant and lies in the mainstream of machine learning. The improvement in the asymptotic bound on the number of features is substantial and the theoretical content looks solid. The mathematical results are presented in detail, with all their assumptions and subtleties made explicit and verifiable.  The authors highlight that the proposed method does not yield a projection of the kernel space, but an operator which is asymptotically close to a projection in a formal sense. However it is not clear how to interpret it, and some discussion could help. What guarantees do we have on the output operator? Do other methods (except ERM) have a similar limitation? How does this discrepancy affect the downstream uses of KPCA? This question arises in light of the experimental results, where the proposed methods seem to produce solutions which are apparently far from being feasible (for eq.(2)) whenever they don't coincide with the baselines (more on this later).  The mathematical part (which forms the bulk of the paper) somewhat lacks in clarity. There is a lot of heavy notation introduced, and some of it seems overly cumbersome and possibly too general for the needs of the paper. An effort to simplify it to the extent possible would help improve readability. The text does not survey the proof techniques or distinguish a novel technical insight from the mundane derivations. This makes it difficult to gauge the level of insight and sophistication of the ideas used in the proofs. A roadmap of the proof or an explicit survey of the claimed "crucial and novel insights" (line 76) would help here, especially since they are claimed to be of potentially more general use; it is currently not evident what they are. On a related note, a comparison with the techniques of prior works on the subject (eg. [RR17] which is cited as a precursor) might be in order, if relevant.  Some more minor clarity issues: 1. The notion of streaming KPCA is stated in the title but not properly explained or referenced in the text. 2. Table 1 is written in terms of epsilon while the text is parameterized by n,m. This epsilon is not defined or explained, even though it is mostly understood by context. Also the table does not address the limitation of the listed methods (apparently, as per above, some of them do not formally solve the KPCA problem as formulated in eq.(2)).  I found the experimental section quite puzzling on several counts. I might be misunderstanding it, and I hope the authors somewhat clarify this in their response. -- You write that ERM is slower by an order of magnitude compared to the other three methods; where do you see this? The running time plots look like all four methods converge within almost exactly the same time, and that ERM is even slightly better than Nystrom. The proposed methods RF-ERM and RF-Oja begin to produce data points much earlier, but it is not at all clear what this means: their objective value is much higher than the maximum, which apparently implies that they are pretty far from being projections, and as written above I am not sure how to interpret this relaxed notion of solution and whether it is usable as output. -- What are we supposed to see in the iteration plots? In what sense are the curves comparable? Do they measure the number of data point samples until convergence? And don't they show (in whatever metric they measure) that RF-Oja is significantly inferior to the two baselines (while RF-ERM is just slightly better)? -- Why are the proposed methods compared only to baselines and not to competitive prior work (such as RF-DSG from Table 1)? -- In general, is the purpose of this section to make a case for the usability of the proposed methods, or just to verify the theoretical predictions of Corollary 4.4?  === CONCLUSION === Pros:  1. The paper deals with an important and fundamental problem, and seems to make solid progress on its theoretical understanding. 2. The mathematical analysis is detailed, elaborate and well-formalized.  Cons: 1. The paper could be improved in clarity of presentation of the mathematical content. The current form might inhibit its accessibility to wider NIPS audience. The results are clearly stated but the techniques are not as well explained. 2. The relaxed notion of solution used by the proposed methods (which is an apparent limitation) could be better discussed in terms of its meaning and downstream applicability. 3. The experimental section is not entirely clear, and in particular does not make a solid case for the empirical usability of the proposed method.  === UPDATE AFTER AUTHOR RESPONSE === Thank you for the clarifications. I would encourage to include some of them in the paper, especially w.r.t the experimental section which became much clearer to me. All suggestions on presentation are left to the authors' discretion; the one I would strongly promote is mentioning the assumptions/limitations of the listed methods in Table 1, so as to maintain formal correctness in the comparison to prior work.


Efficient Anomaly Detection via Matrix Sketching
We consider the problem of finding anomalies in high-dimensional data using popular PCA based anomaly scores. The naive algorithms for computing these scores explicitly compute the PCA of the covariance matrix which uses space quadratic in the dimensionality of the data. We give the first streaming algorithms that use space that is linear or sublinear in the dimension. We prove general results showing that any sketch of a matrix that satisfies a certain operator norm guarantee can be used to approximate these scores. We instantiate these results with powerful matrix sketching techniques such as Frequent Directions and random projections to derive efficient and practical algorithms for these problems, which we validate over real-world data sets. Our main technical contribution is to prove matrix perturbation inequalities for operators arising in the computation of these measures.

Authors develop two new algorithms to find anomalies in high dimensional data when the row data is streaming in. Those algorithms are both two-pass algorithms to approximately compute the rank k leverage score as well as rank k projection distance with running complexity time of O(nd\ell), where n and d are the size of an input matrix A and \ell is independent of n and d, depends only on rank k and some other parameters. While the first algorithm requires O(d\ell) memory, the second one uses only O(log(d)) memory. However, the cost of achieving such reduction is losing accuracy as the first algorithm posses pointwise guarantees from row space approximation and the second one holds average-case guarantees from columns space approximation. The approximation rates are derived analytically and illustrated numerically.   In general, the paper is well-written, well-referenced throughout, and in appropriate format/structure. The practical algorithms presented in this paper contributes to the existing literature by proposing two simple methods which do work for any sketch matrix satisfying certain inequalities ((1) and (2)). The paper can be considered for publication in NIPS 2018 Proceedings if the authors can address the following minor comments in the manuscript:  1. Line 145: Replace "other parameters" with "k and \epsilon".  2. Line 184: Replace "2" with "(2)".  3. Line 187: Remove "for" before "e.g.".  4. Line 189: Replace "a n\times n" with "an n\times n". 5. Line 191: Replace "a \ell\times \ell" with "an \ell\times \ell".   6. Line 198: There is a typo in "psedorandomness".  7. Line 230: To keep the statement of Algorithm 1 consistent with the numerical results, replace "Frequent Directions" in Input of this algorithm with "Random Column Projections". Accordingly, lines 156-157 as well as Footnote 2 should be modified.  8. Figures 2-4: In all six plots, replace "Column Projection" and "Row Projection" with "Algorithm 1" and "Algorithm 2", respectively.   At the end end, I have a minor question: To avoid degeneracy in the spectrum of matrix A in theorems 1 and 3, should we assume that that A is (k,\delta) and (k-1,\delta)-separated?


GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing input tokens and a softmax layer for generating output tokens. For problems with a very large vocabulary size, the embedding and the softmax matrices can account for more than half of the model size. For instance, the bigLSTM model achieves great performance on the One-Billion-Word (OBW) dataset with around 800k vocabulary, and its word embedding and softmax matrices use more than 6GBytes space, and are responsible for over 90% of the model parameters. In this paper, we propose GroupReduce, a novel compression method for neural language models, based on vocabulary-partition (block) based low-rank matrix approximation and the inherent frequency distribution of tokens (the power-law distribution of words). The experimental results show our method can significantly outperform traditional compression methods such as low-rank approximation and pruning. On the OBW dataset, our method achieved 6.6 times compression rate for the embedding and softmax matrices, and when combined with quantization, our method can achieve 26 times compression rate, which translates to a factor of 12.8 times compression for the entire model with very little degradation in perplexity.

The sizes of embedding matrices in NLP tasks have long posed difficult computational problems, either from the inefficiency of operating (softmaxing) over them, or often from the sheer difficulty in storing them.  In this paper the authors take on the latter problem, introducing a method of using multiple low-rank approximations to reduce the size of these matrices.  They rely on frequency binning — the same observation underlying the hierarchical softmax solution to the former problem — to group words, prioritizing the most frequent words to receive higher rank approximations.  This itself leads to significant compression rates with little loss in accuracy, and when further combined with quantization, yields large reduction in memory.  Importantly quantization appears to play nicely with their methodology, and the combined seem to provide much smaller models overall, while performing at least as well as naive quantiziation on large data sets.  The method is simple but effective, and the paper is clearly written, the problem of obvious importance, so my main critique comes from _how_ the material is presented.  (1)  I thought some of the comparisons were more arbitrary than I’d like.  The size of the parameter reduction vs. performance is, of course, a spectrum.  So, for instance, to present a 40x reduction on small LSTM, when the perplexity hit was huge, is not so informative IMO.  It’s already a small model — it’s less interesting to see it reduced 40x vs. the larger ones.  I’d be more interested to see how much it can be reduced while retaining approximately full performance level.  This could be said for many of the presented results — they all seem arbitrarily chosen and while some provide clear take-aways, these are very small glimpses into the larger size-accuracy trade-off at hand.  I’m even tempted to say that, despite what appear to be very compelling results in specific cases on an important problem, and a technique with great practical importance, I’d rather see it resubmitted with more attention paid to painting a clear picture of these trade-offs.  (2)  Some things were over-stated.  Biggest offender: "On the OBW dataset, our method achieved 6.6x compression rate for the embedding and softmax matrices, and when combined with quantization, our method can achieve 26x compression rate without losing prediction accuracy.”  In this setting perplexity goes from 31.04 to 32.47.  I don’t want to claim this is going to make any practical difference, but I don’t understand the need to overstate the effectiveness of this approach when it already does so well.  This is easy to fix so please ammend strong but unsupported claims.  Question for the authors:  If perplexity is all one cares about, and each token is considered uniformly important, than such frequency binning here is a well-motivated decision.  Still, I wonder for many domains where many of the most frequent words are not so semantically interesting, if this will have a negative effect on end-task performance.  Perhaps something like speech recognition, or other product models, where the most frequent words are seen often by the core model and the language model provides little useful information then about them, but now provides poorer approximations to rarer words the core model is also likely to misidentify.  Have the authors looked into a more qualitative error analysis of where the reduced models goes wrong?  Or along these lines, is there anything to say about the post-GroupReduce cluster membership?  Are there any conceptual trends in how the goal of reducing reconstruction error adjusted the freq-based initial partitionings?  Can you comment on the computation time of these experiments?  Is there a noticeable performance difference on compressing just the embedding matrix vs. just the output matrix?  I’m curious if the compression “gains” come from both about equally, or are coming primarily from one or the other.   Misc:  "One word’ rank is" --> typo  Table 2 needs a label of its metric within the figure  "The overall algorithm is in Algorithm 1. Figure 2 illustrates our overall algorithm." —> confusing  "due to the lack of publicly released model checkpoint." —> grammar  "2-LAYER LSTM-8192-1024” —> this model is not shown as mentioned  AVp as AVp ≈ Up(V p)T —> not sure if this is a type or odd phrasing, but could ditch the “as AVp"


MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
Deep learning models exhibit state-of-the-art performance for many predictive healthcare tasks using electronic health records (EHR) data, but these models typically require training data volume that exceeds the capacity of most healthcare systems. External resources such as medical ontologies are used to bridge the data volume constraint, but this approach is often not directly applicable or useful because of inconsistencies with terminology. To solve the data insufficiency challenge, we leverage the inherent multilevel structure of EHR data and, in particular, the encoded relationships among medical codes. We propose Multilevel Medical Embedding (MiME) which learns the multilevel embedding of EHR data while jointly performing auxiliary prediction tasks that rely on this inherent EHR structure without the need for external labels. We conducted two prediction tasks, heart failure prediction and sequential disease prediction, where MiME outperformed baseline methods in diverse evaluation settings. In particular, MiME consistently outperformed all baselines when predicting heart failure on datasets of different volumes, especially demonstrating the greatest performance improvement (15% relative gain in PR-AUC over the best baseline) on the smallest dataset, demonstrating its ability to effectively model the multilevel structure of EHR data.

The authors present a novel methodology designed to take into account some of the structure in the way EHR data is emitted. They observe that a visit consists of a set of diagnosis, which in turn trigger a series of medications and treatments. They leverage this hierarchy to construct multi-level embeddings that allows them to aggregate information across different levels to create embeddings. Overall, I like the idea of using “hidden” structure to improve a model and I like the idea of using the auxiliary loss function.   The authors evaluate all methods to predict heart failure under a variety of scenarios. They mention a sequential disease prediction task in the abstract, but I was unable to find an evaluation for this in the main text, though it is mentioned in the supplement. Without commenting on the results for this task in the main the text, I would recommend removing the reference to it.  On heart failure, they show that their method improves up Med2Vec and GRAM, two existing methods in this area, especially when data are scarce (Figure 3) and when the visit complexity is high (Table 3).  Quality: Over all the paper appears to be sound, though some of the details are missing. I wasn’t clear from the main text or the supplement which version of MiME was the one reported in the main body of the text. Was it the one which included the MLP? If so, as the authors note, this gives MiME and GRAM an advantage of Med2Vec and the fair comparison would add an MLP to Med2Vec. Also, the performance advantage for MiME is less clear when considering the auROC, even though I mostly agree with the authors use of AUC-PR for this purpose.   I also had a question with respect to the data source. The authors state: "We conducted all our experiments using EHR data provided by a large healthcare system" More details about the population in this EHR would help the reader assess the generalizability of this data. Is it US? European? What is the socio-economic status of the population? Ethnicity? etc  I also know that healthcare data can be hard to come by, but 30,000 patients is not large by any standard. Since most methods performed the same when using all of the data, I am unsure if the results seen by the authors would hold when using an EHR of millions of patients, which is common for most major healthcare centers and insurance claims databases.  Clarity: The manuscript is mostly well written, though it is a little hard to follow in spots, mainly in section 2, though I found figure 2 to be very helpful.   Small typos: Page 2: In a typical visit,a physician gives a 59 diagnosis to a patient and then order medications or procedures based on the diagnosis.  Originality: The method presented here is novel and interesting.   Significance: The significance of this work could be improved by using addition datasets (which I know in healthcare is tough) and by considering more conditions. As written, it is of limited significance given the small, elderly population and the single prediction task.


The Price of Fair PCA: One Extra Dimension
We investigate whether the standard dimensionality reduction technique of PCA inadvertently produces data representations with different fidelity for two different populations. We show on several real-world data sets, PCA has higher reconstruction error on population A than on B (for example, women versus men or lower-versus higher-educated individuals). This can happen even when the data set has a similar number of samples from A and B. This motivates our study of dimensionality reduction techniques which maintain similar fidelity for A and B. We define the notion of Fair PCA and give a polynomial-time algorithm for finding a low dimensional representation of the data which is nearly-optimal with respect to this measure. Finally, we show on real-world data sets that our algorithm can be used to efficiently generate a fair low dimensional representation of the data.

The manuscript proposes a dimensionality reduction method called “fair PCA”. The proposed study is based on the observation that, in a data model containing more than one data category (“population” as called by authors), the projection learnt by PCA may yield different reconstruction errors for different populations. This may impair the performance of machine learning algorithms that have access to dimensionality-reduced data obtained via PCA.   To address this problem, the authors propose a variant of the PCA algorithm that minimizes the total deviation between the error of the learnt projection and the error of the optimal projection for each population.  Quality: The paper is based on an interesting idea with an interesting motivation. The technical content of the paper is of satisfactory depth. The proposed work can find usage in many applications.  A few comments and questions:  - A setting with only two populations is considered in the theoretical results and the proposed algorithm. Is it possible to extend these results to a setting with multiple populations? In particular, is it easy to suggest a multiple-population version of Theorem 4.5?  - The experimental evaluation of the proposed algorithm is somewhat limited. It would be very interesting to see whether the proposed dimensionality reduction method improves the performance of machine learning algorithms when used for preprocessing the data containing two/multiple populations.  - Please have a complete pass over the paper to check the language. There are several sentences with typos and mistakes.     Clarity: The presentation of the study is of satisfactory clarity in general. However, please address the following issues:  - The discussion in page 2, lines 62-69 is hard to follow. It would be good to paraphrase with more clear wording.  - Please define the notation [A/B] in line 5 of Algorithm 1.  - In the first plot in Figure 2, the loss seems to increase with the number of dimensions. This contradicts the expectation that the loss should decay to 0 in the long term as the number of dimensions increases, since the approximation error of both PCA and fair PCA would approach 0. How do the authors explain this behavior?   Originality: As far as I know, the work is based on an original idea and problem formulation.  Significance: The proposed study has solid content and may find useful application in many machine learning problems.


Neuronal Capacity
We define the capacity of a learning machine to be the logarithm of the number (or volume) of the functions it can implement. We review known results, and derive new results, estimating the capacity of several neuronal models: linear and polynomial threshold gates, linear and polynomial threshold gates with constrained weights (binary weights, positive weights), and ReLU neurons. We also derive some capacity estimates and bounds for fully recurrent networks, as well as feedforward networks.

Overall assessment: I consider this work to be interesting and worth accepting, even though I am not convinced by the presentation itself, which I feel is obfuscating the real point of the paper. (More details to follow.)  ----------------------------------------------------------------------------- Details: In this work, the author(s) consider, under a specific angle, the following quite fundamental question: "what is the expressive power of a given class C of functions (concepts)?" A natural (though maybe arguable, and not suitable for *all* purposes) is to consider the logarithm of its size, log_2 |C| (or, if C is infinite, the log of its measure, for a well-chosen notion thereof; or the size of a minimum cover). Indeed, this is an approach well-studied in information theory and statistics, for instance (see the related notions of VC dimension, covering number, bracketing entropy, etc.). With this in mind, the paper seeks to obtain tights bounds for classes C of Boolean functions (f: {0,1}^n \to {0,1}) such as (i) linear threshold functions (LTF, halfspaces) (ii) polynomial threshold functions (PTF), (iii) restrictions thereof (small weights, positive weights), and (iv) networks obtained by combining them.  Now, this is indeed interesting. My main point is that these questions are quite hard, and as far as I can tell the results for (iii), and (iv) are quite weak or straightforward. The main contribution of the paper is Theorem 3.1, which provides and actual asymptotically tight bound for (ii), the class of degree-d PTFs  (concluding a decades-long line of work, the previous bounds still being off by a factor d). More specifically:  - Theorem 1 is a preliminary, and should be either a fact or a lemma - Theorem 3.1 is the main technical result, non-trivial, and (in my opinion) should be highlighted in the introduction as the key result - Theorem 4.1 is quite straightforward and easy; and only deals with (binary-weights) LTFs, not PTFs - Theorem 4.2 is not tight; and only deals with (positive-weights) LTFs, not PTFs - I am not able to really quantify the relevance of Theorems 5.1 and 5.2.  In summary, again: I feel the presentation distracts from the main result (Theorem 3.1), promising instead results about more general classes (degree-d PTFs, networks of PTFs and LTFs, more general types of functions...) which end up being unaddressed or for which the results are much less impressive (as in, they are not tight, or are simple corollaries). This is a bit disappointing, since Theorem 3.1 is very good by itself.  (Ironically, this is exactly what the supplemental material is: a math paper focusing entirely on, and dedicated to, Theorem 3.1)  ----------------------------------------------------------------------------- More detailed comments (technical, and typos) below.  - I am really not convinced Figure 1 is useful in any way to understand the paper. - l.41 Misleading. Phrased as it is, it sounds like this problem is still open, while (as you mention later) it has been settled since '89. - l.60 and everywhere: "i.e. " -> "i.e., " - eq (10) (l.89): "leq" -> "\leq" - Section 3.1 (e.g., l.99): you should also discuss Chow's theorem, which gives such an interpretation and upper bound, as it states that any LTF is uniquely determined by its n+1 first Fourier coefficients. Now, since for a Boolean function each Fourier coefficient as granularity 1/2^{n-1}... (see e.g. discussion after Theorem 2.2 in https://arxiv.org/abs/1511.07860) - l.104: similar: you should discuss the degree-d PTF extension of Chow's theorem, due to Bruck [1990]: see e.g. discussion at the end of https://arxiv.org/abs/1206.0985 - Section 4.1: for the sake of the literature review, mention that binary-weight LTFs are commonly referred to as "signed majorities". Also, this section is quite misleading. You start by discussion binary-weight PTFs (degree d, for general d), but prove essentially nothing (except a possibly very loose easy upper bound) for d>1; only considering d=1 (Theorem 4.1). Moreover, a really relevant and important case (as you mention!) is the case of bounded weights, say when the weights are constrained to be poly(n) (see e.g. https://arxiv.org/abs/1508.03061 (STOC'17) and references within). Can you say anything for that case? - l.147: trailing period. - Theorem 4.2: where was T[N,1] introduced? I couldn't find that notation -- it looks like you are using it in lieu of C_1[N,1]?  Moreover, I don't buy the "As a result" statement (and the "reduced by a linear factor" which follows). You have N^2(1+o(1)), so that the second order term you neglect is o(N^2). Since N is also o(N^2), you *cannot* write and discuss this -N term! It could be completely cancelled or drowned by the o(N^2) (which, as far as the statement of the theorem promises, cound be anything, say N^{1.999999}) - ll.178-179: why is it a reasonable thing to do? Detail. - Eq. (17) "\approx =" -> "\approx" Also, more generally (Eq. 16, Eq. 19, Theorem 5...), you never have defined what \approx means for you. Is it informal? Is it notation for the Landau asymptotic \sim? You need to explain, as otherwise one cannot make sense of it. For instance, in (17) using the Landau asymptotic equivalence (which only "guarantees" anything meaningful for the first dominating term) would be strange, as you give 3 terms. Do you mean instead "= N^2 +N + o(N)"? "= N^2 +N + O(1)"? "= N^2 +N - 1 +o(1)" (which given the fact that sizes are integers, would lead to an actual exact equality...)?  In short: you are abusing asymptotic notation here, giving too many terms (leading to a mistaken feeling of precision) l.210: typo ("threshold") l.236: typo ("layered") l.272: typo (ref [17]): "O'Donnell"  Again, the section title and introduction promise results about degree-d PTFs, but the only actual result is for d=1.


Learning to Reason with Third-Order Tensor Products
We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data. This improves symbolic interpretation and systematic generalisation. Our architecture is trained end-to-end through gradient descent on a variety of simple natural language reasoning tasks, significantly outperforming the latest state-of-the-art models in single-task and all-tasks settings. We also augment a subset of the data such that training and test data exhibit large systematic differences and show that our approach generalises better than the previous state-of-the-art.

Summary    This paper presents a question-answering system based on tensor   product representations. Given a latent sentence encoding, different   MLPs extract entity and relation representations which are then used   to update an tensor product representations of order-3 and trained   end-to-end from the downstream success of correctly answering the   question. Experiments are limited to bAbI question answering, which is   disappointing as this is a synthetic corpus with a simple known   underlying triples structure. While the proposed system outperforms   baselines like recurrent entity networks (RENs) by a small difference   in mean error, RENs have also been applied to more real-world tasks   such as the Children's Book Test (CBT).   Strengths    - I like that the authors do not just report the best performance of     their model, but also the mean and variance from five runs.   - I liked the ablation study, showing which operations are needed for     which bAbI tasks.   Weaknesses    - Results on bAbI should be taken with a huge grain of salt and only     serve as a unit-test. Specifically, since the bAbI corpus is     generated from a simple grammar and sentence follow a strict triplet     structure, it is not surprising to me that a model extracting three     distinct symbol representations from a learned sentence     representation (therefore reverse engineering the underlying     symbolic nature of the data) would solve bAbI tasks. However, it is     highly doubtful this method would perform well on actual natural     language sentences. Hence, statements such as "trained [...] on a     variety of natural language tasks" is misleading. The authors of the     baseline model "recurrent entity networks" [12] have not stopped at     bAbI, but also validated their models on more real-world data such     as the Children's Book Test (CBT). Given that RENs solve all bAbI     tasks and N2Ns solve all but one, it is not clear to me what the     proposed method adds to a table other than a small reduction in mean     error. Moreover, the N2N baseline in Table 2 is not introduced or     referenced in this paper, so I am not sure which system the authors     are referring to here.  Minor Comments    - L11: LSTMs have only achieved on some NLP tasks, whereas traditional     methods still prevail on others, so stating they have achieved SotA     in NLP is a bit too vague.   - L15: Again, too vague, certain RNNs work well for certain natural     language reasoning tasks. See for instance the literature on natural     language inference and the leaderboard at     https://nlp.stanford.edu/projects/snli/   - L16-18: The reinforcement learning / agent analogy seems a bit     out-of-place here. I think you generally point to generalization     capabilities which I believe are better illustrated by the examples     you give later in the paper (from lines 229 to 253).   - Eq. 1: This seems like a very specific choice of combining the     information from entity representations and their types. Why is this     a good choice? Why not keep the concatenation of the kitty/cat outer     product and the mary/person outer product? Why is instead the     superposition of all bindings a good design choice?   - I believe section four could benefit from a small overview figures     illustrating the computation graph that is constructed by the     method.   - Eq. 7: At first, I found it surprising why three distinct relation     representation are extracted from the sentence representation, but     it became clearer later with the write, move and backling     functions. Maybe already mention at this point why the three     relation representations are going to be used for.   - Eq. 15: s_question has not been introduced before. I imagine it is a     sentence encoding of the question and calculated similarly to Eq. 5?   - Eq. 20: A bit more details for readers unfamiliar with bAbI or     question answering would be good here. "valid words" here means     possible answer words for the given story and question, correct?   - L192: "glorot initalization" -> "Glorot initialization". Also, there     is a reference for that method: Glorot, X., & Bengio, Y. (2010,     March). Understanding the difficulty of training deep feedforward     neural networks. In Proceedings of the thirteenth international     conference on artificial intelligence and statistics (pp. 249-256).   - L195: α=0.008, β₁=0.6 and β₂=0.4 look like rather arbitrary     choices. Where does the configuration for these hyper-parameters     come from? Did you perform a grid search?   - L236-244: If I understand it correctly, at test time stories with     new entities (Alex etc.) are generated. How does your model support     a growing set of vocabulary words given that MLPs have parameters     dependent on the vocabulary size (L188-191) and are fixed at test     time?   - L265: If exploding gradients are a problem, why don't you perform     gradient clipping with a high value for the gradient norm to avoid     NaNs appearing? Simply reinitializing the model is quite hacky.   - p.9: Recurrent entity networks (RENs) [12] is not just an arXiv     paper but has been published at ICLR 2017.


Direct Runge-Kutta Discretization Achieves Acceleration
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method. When the function is smooth enough, we show that acceleration can be achieved by a stable discretization of this ODE using standard Runge-Kutta integrators. Specifically, we prove that under Lipschitz-gradient, convexity and order-(s + 2) differentiability assumptions, the sequence of iterates generated by discretizing the proposed second-order ODE converges to the optimal solution at a rate of O(N −2 s s+1 ), where s is the order of the Runge-Kutta numerical integrator. Furthermore, we introduce a new local flatness condition on the objective, under which rates even faster than O(N −2 ) can be achieved with low-order integrators and only gradient information. Notably, this flatness condition is satisfied by several standard loss functions used in machine learning. We provide numerical experiments that verify the theoretical rates predicted by our results.

The author analyze the convergence of the RK method applied to a modified version of the "Nesterov Flow" from Su et al. Using a "local flatness" assumption, they show sub-optimal but accelerated rates of convergence in function of 1) the degree of flatness and 2) the order of integration.  Clarity ======= The paper is very clear, all the assumptions and modifications to the differential equation of Su et al., as well as the Lyapunov function, are well motivated. Also, the paper is a very good introduction to Runge Kutta methods.  Small detail, in section 2.1. : Using "S" for the number of stages and "s" for the order of integration is confusing.  In the experiments, by "iteration" you mean the number of gradient calls or the number of "RK" calls?  General =======  - Direct acceleration using *any* RK method is a very good point: The analysis is very generic.  - The assumption of local flatness up to "s" is a bit strong. Do you think it is possible to weaken that assumption?  - In practice, high-order RK methods are quite costly and unstable (because it cancels Taylor coefficients up to "s", but you do not have any control on the coefficient of order "s+1"). Usually, people who uses these methods don't go too far in the number of stages.  - Also, high-order runge kutta methods usually needs an incredibly high number of internal stages (for example, RK of order 10 requires 17 stages). Do you takes the number of internal stages into account in you complexity bound? Source: "A Runge-Kutta Method of Order 10", Ernst Hairer.  - I have a remark about the choice of the step-size. It looks that the computation of the step "h" is not easy. However, there exist adaptive ways to choose it, based on difference between two RK methods with different orders (ex. RK45, which is implemented in Matlab under the name of ODE45). Could you think this can be used as adaptive stepsize in you experiment?  - There exist also other kind of methods: * Linear multi-step methods * Predictor-corrector method * More generally, RK and linear multi steps are instances of "General linear methods" (See the paper "General linear methods" by J. Butcher for instance). Do you think you analysis can be extended to these other standard methods?  - I have a remark about the order of the RK method. In your analysis, you said that you can approach the optimal rate of convergence by increasing the order of the method. However, I think design high-order runge kutta is quite painful in practice (for example, RK10 requires to generates, then solve, a system of 1000+ equations). Do you think is required in practice (in particular, when you use adaptive step size)?  Decision ========  accept (7/10). I particularly like the fact that *direct* discretisation leads to accelerated rate, but the paper lacks a bit of impact. As the conclusion says:  this does not (yet) fully explain acceleration. However, I think this is a good step toward it. The paper presents a promising research direction: since RK-like optimization schemes are not really popular, this may leads to new method and new kind of analysis. I recommend acceptance.   Post-rebuttal ----------------  I read the rebuttal.


Exact natural gradient in deep linear networks and application to the nonlinear case
Stochastic gradient descent (SGD) remains the method of choice for deep learning, despite the limitations arising for ill-behaved objective functions. In cases where it could be estimated, the natural gradient has proven very effective at mitigating the catastrophic effects of pathological curvature in the objective function, but little is known theoretically about its convergence properties, and it has yet to find a practical implementation that would scale to very deep and large networks. Here, we derive an exact expression for the natural gradient in deep linear networks, which exhibit pathological curvature similar to the nonlinear case. We provide for the first time an analytical solution for its convergence rate, showing that the loss decreases exponentially to the global minimum in parameter space. Our expression for the natural gradient is surprisingly simple, computationally tractable, and explains why some approximations proposed previously work well in practice. This opens new avenues for approximating the natural gradient in the nonlinear case, and we show in preliminary experiments that our online natural gradient descent outperforms SGD on MNIST autoencoding while sharing its computational simplicity.

Update: I have considered the author response. I maintain my assessment.  ==========================================================  This paper proposed a convergence analysis on natural gradients. The authors show that the natural gradient updates converge exponentially fast for the deep linear networks with no activation functions and no bias terms. To make the updates faster, the authors proposed some approximation techniques as well as an online implementation. The authors also analyzed the Fisher information structure for the deep linear networks and argue that the block-diagonal style approximations [2, 3, 4, 5, 6] to the inverse Fisher information matrix may ignore the effects of the off-diagonal blocks during the updates. Meanwhile, through analyzing the deep linear networks, the authors explained that the off-diagonal terms contribute the same as the diagonal terms in the updates, and hence explain why block-diagonal approximations may work well in practice.  The problem addressed by this paper is quite interesting. Analyzing the convergence rates of natural gradient methods for general deep neural networks is an open problem, to best of my knowledge. Also, the analysis on the block-diagonal approximation style methods is interesting since these methods are widely applied by the machine learning community. However, I am not sure if the convergence rate analysis of the deep linear networks is useful or not since these networks may not be useful in practice while the case of general deep neural networks may be quite different with this special case.  Strengths of the paper:  1. In my opinion, the research direction of studying the convergence rates of natural gradient updates is interesting and useful, since natural gradients are widely used.  2. Using the deep linear networks to analyze the off-diagonal blocks' contributions on natural gradient updates is a very good idea. This analysis helps us understand why block-diagonal style approximations could work.  Weakness:  1. Though the analysis on deep linear networks show us some insights on the convergences rates of natural gradient updates, the case might be very different for general deep neural networks since they contain bias terms and non-linear activation functions, which might make things much more complicated.  2.  For the deep linear networks with layer sizes n_0, ...., n_L, only the case when n_0 * n_L > \sum\limits_{l=0}^{L-1} n_l * n_{l + 1} is useful (otherwise, it is better just to apply a single matrix W, but not a deep linear network). However, for both experiments in Figure 1(A) and Figure 1(B), \sum\limits_{l=0}^{L-1} n_l * n_{l + 1} is much larger than n_0 * n_L. It is better to show more results on the more interesting cases, such as networks with bottleneck layers (e.g. auto-encoders in Figure 1(C)).  3. For the MNIST experiment in Figure 1(C), perhaps some more comparisons are necessary. First, only comparing to SGD is not sufficient. Other first-order methods (e.g. Adams and RMSProp) or some second-order methods (e.g. Hessian-Free optimization [1]) are good potential baseline methods. Second, auto-encoding on MNIST is a well-studied task. It would be great if the authors could compare the results with the other work on this task. Third, the authors tried to compare the efficiency of their method against SGD through number of iterations, but single iteration of their method may take longer than that of SGD. It will be better if the comparison on time is also presented.  Minor comment:  Probably Section 7 is too long. I suggest maybe split it to a section of related work and a section of conclusion is better.   References:  [1] Martens J. Deep learning via Hessian-free optimization. In ICML 2010 Jun 21 (Vol. 27, pp. 735-742).  [2] Heskes, T. (2000). On natural learning and pruning in multilayered perceptrons. Neural Computation, 12(4):881–901.  [3] Povey, D., Zhang, X., and Khudanpur, S. (2014). Parallel training of dnns with natural gradient and parameter averaging. arXiv preprint arXiv:1410.7455.  [4] Desjardins, G., Simonyan, K., Pascanu, R., et al. (2015). Natural neural networks. In Advances in Neural Information Processing Systems, pages 2071–2079.  [5] Martens, J. and Grosse, R. (2015). Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417.  [6] Grosse, R. and Martens, J. (2016). A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573–582.


Isolating Sources of Disentanglement in VAEs
We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate the β-TCVAE (Total Correlation Variational Autoencoder) algorithm, a refinement and plug-in replacement of the β-VAE for learning disentangled representations, requiring no additional hyperparameters during training. We further propose a principled classifier-free measure of disentanglement called the mutual information gap (MIG). We perform extensive quantitative and qualitative experiments, in both restricted and non-restricted settings, and show a strong relation between total correlation and disentanglement, when the model is trained using our framework.

This paper studies the question of why disentangled representations emerge from VAE training, and through the insights derived in this analysis, is able to propose a simple method to obtain even more disentangling in the representations. The fact that beta does not need to be carefully tuned seems like a major advantage. The proposed ideas seem quite insightful/useful, the writing is clear, and the empirical evaluation is thorough and convincing. The authors have been able to shed some light on the murky topic of disentangling, and I believe these results will be of broader interest to the ML community.   All of the issues/improvements I initially wanted to bring up, I found had already been addressed after reading through the paper a few more times.    Minor Comments:  The authors mention there being confusion about the relationship between disentanglement and mutual information between latent codes and the observed data.  Have you considered studying how beta affects this mutual information?  One major problem with VAEs is the posterior collapse phenomenon, which it seems like the beta-TCVAE would only exacerbate, correct?  Perhaps the beta-TCVAE can be combined with methods that increase the mutual information between latent codes and the observed data to avoid posterior collapse, thus ensuring a representation that is both highly disentangled and controllable.  Response to Author Feedback:  I congratulate the authors on a nice paper and recommend they include the reference to "Controllable Semantic Image Inpainting." in their discussion section.  I also recommending including one sentence about the posterior collapse phenomenon somewhere in the text.


Fully Understanding The Hashing Trick
Feature hashing, also known as the hashing trick, introduced by Weinberger et al. (2009), is one of the key techniques used in scaling-up machine learning algorithms. Loosely speaking, feature hashing uses a random sparse projection matrix A : R n → R m (where m n) in order to reduce the dimension of the data from n to m while approximately preserving the Euclidean norm. Every column of A contains exactly one non-zero entry, equals to either −1 or 1. Weinberger et al. showed tail bounds on Ax 2 2 . Specifically they showed that for every ε, δ, if x ∞ / x 2 is sufficiently small, and m is sufficiently large, then
These bounds were later extended by Dasgupta et al. (2010) and most recently refined by Dahlgaard et al. (2017), however, the true nature of the performance of this key technique, and specifically the correct tradeoff between the pivotal parameters x ∞ / x 2 , m, ε, δ remained an open question. We settle this question by giving tight asymptotic bounds on the exact tradeoff between the central parameters, thus providing a complete understanding of the performance of feature hashing. We complement the asymptotic bound with empirical data, which shows that the constants "hiding" in the asymptotic notation are, in fact, very close to 1, thus further illustrating the tightness of the presented bounds in practice.

The authors carry out a fine-grained analysis of locality sensitive hashing, and prove data-dependent bounds in order to provide a better understanding of how much random compression can be applied to data in realistic settings, while still preserving Johnson-Lindenstrauss guarantees on the compressed data. This is useful to know, since existing lower bounds on the projected dimension generally proceed by finding the minimal projection dimension required to compress the worst case (e.g. data lying at the vertices of the \ell_1 ball or some similarly structured scenario) but in practice observed data frequently have much more forgiving structure. The paper is interesting and, as far as I can see, correct although I did not check *all* the details thoroughly.  Although there is nothing really wrong with this paper - and I commend the authors for their work - I do however have two reasons why I think it may not be suitable for NIPS: The first is that I think this work is likely of more interest to a databases, KDD, or theoretical CS audience and therefore NIPS is perhaps not the best venue to gain this work the widest audience of interested readers. The authors clearly feel otherwise, but this is my well-intended advice. The second, more serious, concern I have is that - as a purely theoretical paper - I think that the main contributions here are really the proof techniques in the supplementary material, and I think there is not enough room to include even a reasonable sketch of these within the NIPS page limit. That is, I don't think the contributions in the manuscript are enough for NIPS but, notwithstanding my comment above, I do think that the longer paper in the supplementary paper could have been. Since NIPS papers should be self-contained I think that is a problem.  FWIW I think the supplementary version of this paper is clearly of a publishable standard at a very good venue, and believe the authors would be helping themselves by either disseminating their work as a journal paper or else submitting to a conference with a more generous page limit (e.g. ECML-PKDD 16 pages in LNCS style) or a tighter style (e.g. KDD 9 pages inc. refs. in ACM style).  Please note also that I do not object to this paper because it is theoretical: I think that theory papers are important contributions to all of the serious conferences, and personally I would be happy to see more of them. However in my own experience it is not always easy to balance providing enough detail about how the results were obtained with substantial outcomes in a short conference paper, and here I don't think the authors have found that balance.  Addendum: The authors did not address my comments in rebuttal, but checking post-review I see that the supp mat version is available on the ArXiv. I prefer that version to the NIPS version a lot, and encourage the authors (if they need any encouragement) to link to it from their NIPS submission should it be accepted. Since a guarantee of the details absent in the NIPS version in some archival form was my main concern here, I have accordingly raised my rating to "accept".


Invariant Representations without Adversarial Training
Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation. Unfortunately, learning representations that exhibit invariance to arbitrary nuisance factors yet remain useful for other tasks is challenging. Existing approaches cast the trade-off between task performance and invariance in an adversarial way, using an iterative minimax optimization. We show that adversarial training is unnecessary and sometimes counter-productive; we instead cast invariant representation learning as a single information-theoretic objective that can be directly optimized. We demonstrate that this approach matches or exceeds performance of state-of-the-art adversarial approaches for learning fair representations and for generative modeling with controllable transformations.

The paper studies the timely and extremely relevant problem of learning invariant representations for data. In contrary to the main stream of works in this area, it does not consider an adversarial training approach, and this is refreshing! Instead, it proposes to solve an information-theoretic objective, which proves to be competitive and even better that state-of-the-art adversarial approaches.   In more details, the authors derive a useful upper-bound on the mutual information, that can be nicely integrated in the VAE or VIB frameworks. This derivation is essentially the main contribution of the paper, and it provides an elegant and generic framework for learning invariant representations.   The paper is generally well written, even it is quite dense and probably too detailed at places, like in Section 2. This sort of leads the reader to lose the main intuition behind the proposed framework.   The experiments are generally well chosen, and the visualisation of Fig 2 is pretty interesting and original. The results are however not always fully convincing (see for example the German dataset in Fig 1), but it does not critically reduce the quality of the paper and the proposed ideas.   Suggestions:  - the title 'evading the adversary' is a bit misleading: what is done in the paper is actually learning invariant representations... - while there is an attempt to stay generic in the presentation of the framework, it sometimes leads to confusion: in particular, the mix between fairness ideas, and more classical invariant representation learning objectives is sometimes confusing (like in the related work section, and in the experiments).   Overall, the paper addresses a timely problem, with a new approach, based on information theoretic concepts and bounds on the mutual information. These ideas are integrated into classical learning architectures, and leads to competitive results.


Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks
Real-world image recognition is often challenged by the variability of visual styles including object textures, lighting conditions, filter effects, etc. Although these variations have been deemed to be implicitly handled by more training data and deeper networks, recent advances in image style transfer suggest that it is also possible to explicitly manipulate the style information. Extending this idea to general visual recognition problems, we present Batch-Instance Normalization (BIN) to explicitly normalize unnecessary styles from images. Considering certain style features play an essential role in discriminative tasks, BIN learns to selectively normalize only disturbing styles while preserving useful styles. The proposed normalization module is easily incorporated into existing network architectures such as Residual Networks, and surprisingly improves the recognition performance in various scenarios. Furthermore, experiments verify that BIN effectively adapts to completely different tasks like object classification and style transfer, by controlling the tradeoff between preserving and removing style variations. BIN can be implemented with only a few lines of code using popular deep learning frameworks.

Update following the author rebuttal:  I would like to thank the authors for providing a thoughtful rebuttal and addressing the points I raised in my review.  -----  The paper proposes a method -- termed Batch-Instance Normalization (BIN) -- for automatically learning which of batch (BN) or instance (IN) normalization should be employed for individual feature maps in a convolutional neural network. The justification is simple and intuitive: in certain cases it is beneficial to standardize features on a per-example basis (e.g., in most cases global illumination is irrelevant for object classification), but in other cases doing so is hurtful (e.g., when predicting weather or time global illumination is a very informative feature).  The authors propose a simple idea: batch normalization and instance normalization are blended through a convex combination of both forms of normalization, and the blending coefficient is treated as a model parameter and learned jointly with the rest of the network parameters. I like the simplicity of the proposed method. The paper is well-written, straightforward, and contextualizes the idea well alongside relevant work.  The authors present experimental results for object classification, multi-domain training, and artistic style transfer. For object classification, accuracies and learning curves are presented for ResNet architectures trained on CIFAR10, CIFAR100, and ImageNet, showing that BIN consistently outperforms BN, IN, and BN+IN (i.e., BIN with a constant blending coefficient of 0.5) by a small margin. In particular, the comparison with BN+IN suggests that the adaptive nature of the proposed idea is responsible for the observed improvements, rather than the the ensembling of BN and IN. The authors also analyze the distribution of blending coefficients and show that the distribution is roughly bimodal, with most of the probability mass concentrated near the BN end of the spectrum and a smaller amount concentrated near the IN end of the spectrum, which aligns well with the intuition put forward by the paper. One criticism I have to voice is that the absence of confidence intervals for the presented accuracies makes it hard for me to evaluate whether the improvements presented are statistically significant in some cases. The consistent nature of the improvements -- best exemplified by Table 2 -- goes some way towards building confidence in their statistical significance, but I would nevertheless like to have a sense of what sort of variability to expect due to simple factors like random weight initialization and whether the observed improvements deviate significantly from it.  For multi-domain training, the authors compare BIN to BN in a mixed-domain classification setting. Once again, consistent improvements are shown, and the authors conclude that BIN’s ability to alleviate style disparities between domains is responsible for the increase in accuracies. I believe this is a likely explanation, but I would like to see whether BIN also improves performance when training on individual domains, as it could be that the ability to alleviate intra-domain style disparities is also helpful. The authors also compare BIN to BN in a domain-adaptation setting and show that BIN outperforms BN in most settings by a small margin.  For style transfer, the paper compares BIN with BN, IN, and BN+IN when training a feed-forward style transfer network on a single style image. They conclude that BIN makes a trade-off between the content loss and the style loss (which are found to be respectively slightly lower and slightly higher than their BN counterparts), and that BIN is better than BN+IN at minimizing both the content and style losses. Presenting results on more than one style would help build confidence in the significance of the observed differences; alternatively, presenting results for a multi-style feedforward network like the ones proposed by Dumoulin et al., Huang et al., or Ghiasi et al. (G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, J. Shlens. “Exploring the structure of a real-time, arbitrary neural artistic stylization network”. Proceedings of the British Machine Vision Conference (2017).) would be equally convincing.  In summary, the paper proposes a simple idea, explains it in a straightforward and well-written fashion, and presents thorough and convincing empirical evidence that it improves performance in multiple problem settings. The biggest criticism I have to voice is that I would like to see confidence intervals for the accuracies. Although the idea itself is not groundbreaking, I believe it is a useful and relevant addition to a deep learning researcher’s toolkit.


Multilingual Anchoring: Interactive Topic Modeling and Alignment Across Languages
Multilingual topic models can reveal patterns in cross-lingual document collections. However, existing models lack speed and interactivity, which prevents adoption in everyday corpora exploration or quick moving situations (e.g., natural disasters, political instability). First, we propose a multilingual anchoring algorithm that builds an anchor-based topic model for documents in different languages. Then, we incorporate interactivity to develop MTAnchor (Multilingual Topic Anchors), a system that allows users to refine the topic model. We test our algorithms on labeled English, Chinese, and Sinhalese documents. Within minutes, our methods can produce interpretable topics that are useful for specific classification tasks.

The paper proposes an anchor-based approach for learning a topic model in multilingual settings. The model first is built automatically and then can be refined by interaction with a user. The paper addresses a problem of interest of the previous NIPS submissions.  Quality: The paper is technically sound. I like that for bilingual topic modeling it is only require to have a dictionary between two languages but documents should not be aligned for training a topic model. The interactive part could be interesting for applications but is not quite impressive from the methodological point of view since building a model interacting with a user is based on existing methods where the proposed approach for multilingual anchoring is used only for initial topic learning.   The main missing piece in the paper from the technical point of view is absence of description of main optimisation algorithm. How exactly the authors solve their equation (5) in an efficient way.   Could the authors please explain how do they choose the number of anchoring words if they allow multiword anchoring? Or are multi-words only used in the interactive settings?  It also would be beneficial to see more discussion about conceptual comparison of generative and anchor-based approaches for topic modeling.   Other suggestions/questions: 1. It would be an interesting comparison if MCTA results are provided both for limited and unlimited time scenarios. Now it is unclear whether MCTA gives poor results because it has not converged or because it generally learns poorer topics. 2. Follow-up regarding comparison with MCTA, the authors discuss briefly computational time of the methods without specifying what programming languages and hardware are used for implementations 3. Does the method have to be bilingual? It seems that it can be extended for more languages. If it is so, it could be mentioned in the paper, if the method is limited to be bilingual, it should be clearly stated. 4. I am not sure about user study experiment setup because the users were asked to essentially optimise classification performance of the system rather than interpretability of the topics directly. Interpretability is something that they probably based their changes on but this was not their primal goal. And it could be a case that a user made a change to improve interpretability of topics but this harmed classification performance since classification performance is not a direct measure of interpretability of topics and may not be related to it. As follow-up, the arguments in lines 255-257 about why one should care about interpretable features it is again talked about as classification is the end goal of topic modeling which is not correct since then one should most probably use some deep learning approaches rather than topic modeling 5. Figure 3b. What is the baseline here? Is it MCTA? Or is it multilingual anchoring?  Clarity: The paper is mostly well written. I like a lot of examples used, this is really engage reading. Below are some suggestions for improvement: 1. Figure 2 is unreadable. The plots should be enlarged 2. Figure 3b. It is unclear why the plot type has changed from the one used in Figure 3a. There are lines in Figure 3a, but scatter points in Figure 3b. 3. At the beginning of Section 4.1 it is better to remind a reader that “multilingual anchoring” is the proposed method without human interaction and “MTAnchor” is the proposed method with human interaction 4. Table 1 is missing an actual caption what this table contains. The current caption discusses the results 5. Lines 234-236 specifying the evaluation procedure should be before user study experiment since it uses the same procedure 6. It is better to specify that NPMI is 7. Section 4.1 misses description of results from Table 1. For example, the authors could move the current caption of Table 1 into the main body of the paper. 8. Line 245: “a user can reach at least a 0.40 increase” – since the median results among users are the same as multilingual anchoring should it be “a user can reach UP TO a 0.40 increase”? 9. Agruments in line 255-257  10. Section 6 on Related work should probably be moved before Section 3 as in Section 3 the authors refer to JointLDA and in Sections 4 and 5 the authors compare their method with MCTA, both of which are introduced in Section 6. 11. Last sentence of the paper does not seem to be of any use – a very vague statement.  Minor things: 1. S_k in equation (3) is not defined 2. The sentence “A computationally attractive …” in line 43 appears quite rapidly after general words about topic modeling. Maybe some more introductory words could help to direct the subject towards an anchor word approach 3. Typo: inperfection -> imperfection (line 113) 4. The footnote 3 is better to be stated in the main body of the paper 5. Typo: line 161 “not enough translations may be unavailable”. Should it be “available”? 6. Typo: line 192 “anchors are defined anchors so that they take full control …”. Should it be “users define anchors …”? 7. Typo: line 264 missing articles before “experiments” and “multilingual anchoring algorithm” 8. Typo: line 266 grammatically incorrect sentence. Should it be “approach to modeling cross-lingual topics THAT is both fast and accurate”? 9. Typo: line 267 “between topics if not training time is not long enough” – should first “not” be removed?  Originality: The paper addresses an interesting problem in machine learning. The multilingual anchoring algorithm appears to be fairly novel while interactive MTAnchor seems to be a good contribution only as an application rather than a novel method.  The related work is cited well for generative multilingual models and anchor-based methods, while alternative approaches for multilingual data processing are not discussed such as, e.g., Xiao and Guo, 2013 “A Novel Two-Step Method for Cross Language Representation Learning”  Significance: The paper extends the anchor-based approach for topic modeling for multilingual data and could be a nice contribution in that area. The empirical results on comparison to one generative topic model show superiority of the proposed method, although as mentioned above it is unclear whether it is a matter of restricted time for training or general inability of generative topic model. In any way, the proposed method would have an advantage of faster convergence.   As mentioned above, the interactive method proposed in the paper does not seem to provide a significant contribution.  UPDATE AFTER RESPONSE: I would like to thank the authors for their response and addressing many of the concerns raised by the reviewers. Based on the response I would recommend the authors to focus more in general on a topic modeling (exploratory) point of view and less on classification and use classification accuracy also as a way to show how good is a found topic model, as an indicator of interpretability of topics. Also I would strongly encourage the authors to provide the results of MCTA after its convergence (maybe together with unconverged results) and either claim that the proposed algorithm both is more efficient and finds better topics than MCTA or that it is only more efficient if the results are competitive. The current version with only unconverged MCTA results raises questions and it is not very convincing.


Distributionally Robust Graphical Models
In many structured prediction problems, complex relationships between variables are compactly defined using graphical structures. The most prevalent graphical prediction methods-probabilistic graphical models and large margin methods-have their own distinct strengths but also possess significant drawbacks. Conditional random fields (CRFs) are Fisher consistent, but they do not permit integration of customized loss metrics into their learning process. Large-margin models, such as structured support vector machines (SSVMs), have the flexibility to incorporate customized loss metrics, but lack Fisher consistency guarantees. We present adversarial graphical models (AGM), a distributionally robust approach for constructing a predictor that performs robustly for a class of data distributions defined using a graphical structure. Our approach enjoys both the flexibility of incorporating customized loss metrics into its design as well as the statistical guarantee of Fisher consistency. We present exact learning and prediction algorithms for AGM with time complexity similar to existing graphical models and show the practical benefits of our approach with experiments.

Distributionally Robust Graphical Models  The authors suggest dealing with the structure prediction task using adversarial graphical model (AGM), a generative model trained using an adversary distribution, instead of the empirical one. Instead of focusing on the loss metric, the authors focus on the graphical model allowing more flexibility wrt the loss metric. The AGM algorithm has similar complexity to conditional random fields but is less limited in its loss metric and is Fisher consistent for additive loss metrics.   Following a complex mathematical transformation, the authors provide optimization of node and edge distribution of the graphical model but since this optimization is intractable, they restrict their method to tree-structural models or models with low treewidths. I would expect the authors to discuss this limitation of their algorithm.  I have several reservations regarding the experimental section for which the authors’ may wish to reply. First, only two prediction problems are evaluated; both with low treewhidth, a condition which suits the AGM more than its competitors and does not reflect most prediction problems. Second, different loss metrics are used for each of the problems. Third, while results for the first problem are reported as an average over 20 data splits, no details about the methodology of the second experiment are given. In addition, since no statistical significance was reported for the results of the latter experiment, I assume these results are reported on the entire data set. Fourth, instead of performing the parametric t-test to compare several algorithms on multiple (20) data sets, a non-parametric test (e.g., Friedman’s test with a post-hoc test) should have been exercised. Thus, whether the results in Table 1 are significant or not is due to a suitable test being exercised. Fifth, the AGM shows no advantage over its competitors for the two prediction tasks tested, and thus I would solicit the authors to discuss the method benefits, practicality, and generalizability to non-tree problems.  Major issues  Not all terms are defined (see e.g., Section 2.1 and 2.2)  The authors claim: “This two-stage prediction approach can create inefficiencies when learning from limited amounts of data since optimization may focus on accurately estimating probabilities in portions of the input space that have no impact on the decision boundaries of the Bayesian optimal action” – but this claim is controversial wrt the author selection in a generative model (i.e., graphical models) for prediction. The authors are asked to motivate their selection in the graphical model that required applying complex mathematical transformations in order to make it a prediction model.  Minor issues  56 – argmax vs argmin?  Include directed graphical models (Bayesian networks) in you your survey of probabilistic graphical models to complete it.  References: Capitalize “Bayesian”, conference/journal names, Arc; decide on a consistent use of author and journal names  Quality: good Clarity: average Originality: average Significance: poor  The authors response has raised my opinion about the experimental part of the paper and thus also about the paper itself.


Importance Weighting and Variational Inference
Recent work used importance sampling ideas for better variational bounds on likelihoods. We clarify the applicability of these ideas to pure probabilistic inference, by showing the resulting Importance Weighted Variational Inference (IWVI) technique is an instance of augmented variational inference, thus identifying the looseness in previous work. Experiments confirm IWVI's practicality for probabilistic inference. As a second contribution, we investigate inference with elliptical distributions, which improves accuracy in low dimensions, and convergence in high dimensions.

Update: in light of Reviewer One's observation that Theorem 3 is a special case of Proposition 1(c) of the FVO paper, and that Theorem 1 a version -- after some manipulation -- of Claim 1 in the AESMC paper, I will downgrade my review from a 7 to a 6.  -----------  The paper provides semantics for the posterior approximation that is implicit in Importance Weighted Autoencoders [4].  In particular, their key result is that by maximizing the heuristically motivated importance weighted ELBO (IW-ELBO) from [4] is equivalent to minimizing a particular KL divergence (Theorems 1 and 2).  This KL divergence is not directly between an approximation and the target posterior p(z | x), but contains an additional term that quantifies the effect of importance sampling.  However, this insight motivates the use of the optimizer of the IW-ELBO to compute improved estimates of posterior expectations (equation 9).  The paper is well-written and well-motivated, and provides valuable semantics to the IW-ELBO, and I recommend that it be accepted.  I have some concerns about the the importance weighting procedure, however -- in particular, about whether or not the weights (denoted R) can be expected to have finite variance.  To be fair, these concerns also extend to the original published work [4] to some extent, although [4] skirts the problem somewhat by only concerning itself with log(R) (appendix B of [4]), whereas this paper relies on assertions about R itself.  Specifically, the claim that R_M places less mass near zero for larger M (line 101) does not seem obvious to me in cases where equation (4) has infinite variance.  Similarly, the usefulness of Theorem 3 depends on R having finite variance.  It seems that additional conditions would need to hold to ensure that equation (4) has finite variance -- see Owen [13], example 9.1 and related discussion.  The original reference Burda et al. [4] does not seem to address this problem -- they only control the mean absolute of the E[log R_M], certainly not the variance of R_M.  Note also that Burda et al. [4] Theorem 1 guarantees only that the bound does not degrade, but the authors claim that the bound in fact improves.  Given the tendency of vanilla KL(q || p) to have q very small where p is large, I’d like to see this potential issue addressed more directly.  (At line 165 the authors say that “the variance of R is a well-explored topic in traditional importance sampling”, which is true, but I think it is still important to address the variance of R in this particular context.)  In particular, I wonder whether this method could work when you don’t allow the variational parameterization to have a full covariance matrix (that is, a non-diagonal A_w in equation 14).  Using a full covariance matrix is only feasible in relatively small dimensions because the number of free parameters in A_w scales quadratically with dimension.  Notably, most of the experiments are all relatively low dimensional -- the clutter and dirichlet models use no more than a 10-dimensional parameter space, and even the largest “large” logistic regression model has only up to a 500-dimensional parameter.  (Notably the authors do not report computation time for the logistic model.)   One metric that would be revealing would be the entropy of the normalized weight distribution.  Are a relatively few data points getting most of the weight after normalization?  If so, it’s an indication that the importance sampling procedure is not working well.  A relatively even distribution of normalized weights throughout the optimization process would be reassuring that importance weighting is doing what the authors claim.  It would be more interesting to compare the logistic regression model posterior moments to an MCMC ground truth as with the other models.  Minor typos: - Typo in the definition of R in line 20 -- there should be no log. - The bibliography looks strange.  What are the trailing numbers? -  Line 109: Can be understood - Algorithm 1.  The weight function and q should be explicit or an input.  (I think for theorem 1 to hold they need to be the importance weights of equation 3, and it should be clearer that they are not general.) - Theorem 1. Please define things before you refer to them (specifically p_M) -  Line 184 “y” should probably be “u” - Line 191: It would probably be better to avoid double use of the letter w for both weights and variational parameters - Line 220: gradient -> gradients - I find the notation “E-IWVI” vs “IWVI” (line 231) to be a bit strange.  There doesn’t seem to be anything about the importance weighting trick that requires a Gaussian approximation.  So using the unmodified “IWVI” to refer to a Gaussian q and the modified “E-IWVI” to refer to an elliptical q is confusing.  Perhaps “G-IWVI” could refer to the Gaussian q, and the unmodified “IWVI” can go back to referring to the importance weighting trick irrespective of the choice of q. - Double periods in the caption of figure 5.


Bayesian Distributed Stochastic Gradient Descent
We introduce Bayesian distributed stochastic gradient descent (BDSGD), a highthroughput algorithm for training deep neural networks on parallel computing clusters. This algorithm uses amortized inference in a deep generative model to perform joint posterior predictive inference of mini-batch gradient computation times in a compute cluster specific manner. Specifically, our algorithm mitigates the straggler effect in synchronous, gradient-based optimization by choosing an optimal cutoff beyond which mini-batch gradient messages from slow workers are ignored. The principle novel contribution and finding of this work goes beyond this by demonstrating that using the predicted run-times from a generative model of cluster worker performance improves over the static-cutoff prior art, leading to higher gradient computation throughput on large compute clusters. In our experiments we show that eagerly discarding the mini-batch gradient computations of stragglers not only increases throughput but sometimes also increases the overall rate of convergence as a function of wall-clock time by virtue of eliminating idleness.

========================================Update after authors' rebuttal======================================== I am generally happy with the authors' rebuttal, but would like to add a response to the reply, "For our purposes, we find that batch-size and objective function gain to be correlated in the literature, assuming strong convexity and Lipschitz-continuity ... ". These are rather strong assumptions that are not necessarily true, especially for deep neural networks. Furthermore, finding this optimal m (after taking the product with time-to-m) is non-linearly dependent on the local convexity and Lipschitzness, i.e. you really should be optimizing for objective function gain!  To be clear, the paper remains interesting irrespective of this, but I do think it would be a good direction for future research to properly understand the difference between optimizing throughput versus optimizing objective function gain. ================================================================================   This paper describes a method to improve on Chen et al, by automatically learning and deciding on the number of workers to wait for in distributed synchronous SGD, so as maximize "throughput" and avoid the straggler effect. This is done by using deep linear dynamical models to estimate worker run-times. The number of workers to wait for is then dynamically determined via model inference and maximizing the expected throughput at the next iteration. Experiments conducted show that the method can provide improvements over the static cutoff of Chen et al, and over a baseline dynamic Elfving cutoff.   This work improves over Chen et al by introducing a dynamic cutoff in a principled manner, further improving the state-of-the-art for synchronous SGD. In spite of the seemingly heavy machinery, the experiments show that a dynamic estimation works better than the static cut-off. This work could generate discussion in the systems-ML community, further the debate between asynchronous and synchronous distributed methods.  I have a number of questions / suggestions:  - The technique targets maximization of "throughput", which does not take into account the improvement in the objective function. What we're really interested in is the "rate of objective improvement", or roughly speaking, \Delta(c) / \tilde{x}_{(c)}, where \Delta(c) is the decrease in the objective function using gradients from c workers. Despite the claims (line 112-113) that optimizing throughput "handles tradeoff between iteration speedup and learning signal reduction", I do not see how it addresses the optimization problem / gradient variance. To make this more concrete, consider the optimization of a convex problem. At the start, we would expect all stochastic gradients to be generally in agreement, in which case, it might be best to use few workers, since \Delta may not change much with larger c. As we approach the opt, stochastic noise dominates, and \Delta increases quickly with larger c, so we would want to use as many workers as possible while still mitigating for stragglers.  - In the experiments, comparison between methods is often made in terms of time to reach X iterations. However, the real measure should be time to reach a loss or accuracy target. (After all, the entire point of cutoff methods is to trade-off between the statistical efficiency of synchronous methods and the throughput of asynchronous methods to achieve the best wall-clock time to accuracy, and not the best wall-clock time to target number of iterations.)  - As mentioned above, a lot of work goes into estimating the dynamic cutoff point. How much computation time does this take relative to the gradient computation?  - How does the cutoff point change over the course of the optimization? If it stabilises quickly, can we switch to a static method, or at least reduce the frequency at which the cutoff is changed?  - How would the proposed method compare with a naive approach of constructing an empirical CDF (a la Figure 2) and using that to determine cutoff?