We propose a novel subgraph image representation for classification of network fragments with the target being their parent networks. The graph image representation is based on 2D image embeddings of adjacency matrices. We use this image representation in two modes. First, as the input to a machine learning algorithm. Second, as the input to a pure transfer learner. Our conclusions from multiple datasets are that
• deep learning using structured image features performs the best compared to graph kernel and classical features based methods; and, • pure transfer learning works effectively with minimum interference from the user and is robust against small data.
With the advent of big data, graphical representation of information has gained popularity. Being able to classify graphs has applications in many domains. We ask, "Given a small piece of a parent network, is it possible to identify the nature of the parent network (Figure1)?" We address this problem using structured image representations of graphs.
Adjacency matrices are notoriously bad for machine learning. It is easy to see why, from the unstructured image of a small fragment of a road network, in figure (a) below. Though the road network is structured, the random image would convey little or no information to machine learning algorithms (in the image, a black pixel at position (i, j) corresponds to an edge between nodes i and j). Reordering the vertices (figure (b) below) gives a much more structured image for the same subgraph as in (a). Now, the potential to learn distinguishing properties of the subgraph is evident. We propose to exploit this very observation to solve a basic graph problem (see Figure 1). The datasets mentioned in Figure 1 are discussed in Section 2.4. We stress that both images are lossless representations of the same adjacency matrix. We use the structured image to classify subgraphs in two modes: (i) Deep learning models on the structured image representation as input. (ii) The structured image representation is used as input to a transfer learner (Caffe: see Section 2.3) in a pure transfer learning setting without any change to the Caffe algorithm. Caffe outputs top-k categories that best describe the image. For real world images, these Caffe-descriptions are human friendly as seen in Figure 2a. However, for network-images, Caffe Figure 2: An image of a dog and a structured image of a Facebook graph sample vs their corresponding maximally specific classification vectors returned by Caffe gives a description which doesn't really have intuitive meaning (Figure 2b). We map the Caffedescriptions to vectors. This allows us to compute similarity between network images using the similarity between Caffe description-vectors (see Section 2).

The paper proposed a subgraph image representation and validate it in image classification and transfer learning problems. The image presentation is a minor extension based on a method of producing permutation-invariant adjacency matrix. The experimental results supports the claim.
It is very positive that the figures are very helpful for delivering the information.
The work seems to be a little bit incremental. The proposed image representation is mainly based on a previous work of permutation-invariant adjacency matrix. A novelty of this work seems to be transforming a graph into an image. By the proposed representation, the authors are able to apply image classification methods (supervised or unsupervised) to subgraph classification.
It will be better if the authors could provide more details in the methodology or framework section.
The experiments on 9 networks support the claims that the image embedding approaches with their image representation of the subgraph outperform the graph kernel and classical features based methods. It seem to be promising when using transfer learning.
The last two process figures in 1.1 can be improved. No caption or figure number is provided.
It will be better to make the notations easy to understand and avoid any notation in a sentence without explanation nearby.
For example:
"the test example is correctly classified if and only if its ground truth matches C."(P5)
"We carry out this exercise 4 times and set n to 8, 16, 32 and 64 respectively."(P6)
Some minor issues:
"Zhu et al.(2011) discuss heterogeneous transfer learning where in they use..."(P3)
"Each label vector (a tuple of label, label-probability pairs)." (incomplete sentence?P5)


State-of-the-art deep reading comprehension models are dominated by recurrent neural nets. Their sequential nature is a natural fit for language, but it also precludes parallelization within an instances and often becomes the bottleneck for deploying such models to latency critical scenarios. This is particularly problematic for longer texts. Here we present a convolutional architecture as an alternative to these recurrent architectures. Using simple dilated convolutional units in place of recurrent ones, we achieve results comparable to the state of the art on two question answering tasks, while at the same time achieving up to two orders of magnitude speedups for question answering.

This paper borrows the idea from dilated CNN and proposes a dilated convolution based module for fast reading comprehension, in order to deal with the processing of very long documents in many reading comprehension tasks. The method part is clear and well-written. The results are fine when the idea is applied to the BiDAF model, but are not very well on the DrQA model.
(1) My biggest concern is about the motivation of the paper:
Firstly, another popular approach to speed up reading comprehension models is hierarchical (coarse-to-fine) processing of passages, where the first step processes sentences independently (which could be parallelized), then the second step makes predictions over the whole passage by taking the sentence processing results. Examples include , "Attention-Based Convolutional Neural Network for Machine Comprehension", "A Parallel-Hierarchical Model for Machine Comprehension on Sparse Data", and "Coarse-to-fine question answering for long documents"
This paper does not compare to the above style of approach empirically, but the hierarchical approach seems to have more advantages and seems a more straightforward solution.
Secondly, many existing works on multiple passage reading comprehension (or open-domain QA as often named in the papers) found that dealing with sentence-level passages could result in better (or on par) results compared with working on the whole documents. Examples include "QUASAR: Datasets for question answering by search and reading", "SearchQA: A new q&a dataset augmented with context from a search engine", and "Reinforced Ranker-Reader for Open-Domain Question Answering". If in many applications the sentence-level processing is already good enough, the motivation of doing speedup over LSTMs seems even waker.
Even on the SQuAD data, the sentence-level processing seems sufficient: as discussed in this paper about Table 5, the author mentioned (at the end of Page 7) that "the Conv DrQA model only encode every 33 tokens in the passage, which shows that such a small context is ENOUGH for most of the questions".
Moreover, the proposed method failed to give any performance boost, but resulted in a big performance drop on the better-performed DrQA system. Together with the above concerns, it makes me doubt the motivation of this work on reading comprehension.
I would agree that the idea of using dilated CNN (w/ residual connections) instead of BiLSTM could be a good solution to many online NLP services like document-level classification tasks. Therefore, the motivation of the paper may make more sense if the proposed method is applied to a different NLP task.
(2) A similar concern about the baselines: the paper did not compare with ANY previous work on speeding up RNNs, e.g. "Training RNNs as Fast as CNNs". The example work and its previous work also accelerated LSTM by several times without significant performance drop on some RC models (including DrQA).
(3) About the speedup: it could be imaged that the speedup from the usage of dilated CNN largely depends on the model architecture. Considering that the DrQA is a better system on both SQuAD and TriviaQA, the speedup on DrQA is thus more important. However, the DrQA has less usage of LSTMs, and in order to cover a large reception field, the dilated CNN version of DrQA has a 2-4 times speedup, but still works much worse. This makes the speedup less impressive.
(4) It seems that this paper was finished in a rush. The experimental results are not well explained and there is not enough analysis of the results.
(5) I do not quite understand the reason for the big performance drop on DrQA. Could you please provide more explanations and intuitions?


Ensembling multiple predictions is a widely-used technique to improve the accuracy of various machine learning tasks. In image classification tasks, for example, averaging the predictions for multiple patches extracted from the input image significantly improves accuracy. Using multiple networks trained independently to make predictions improves accuracy further. One obvious drawback of the ensembling technique is its higher execution cost during inference.This higher cost limits the real-world use of ensembling. In this paper, we first describe our insights on relationship between the probability of the prediction and the effect of ensembling with current deep neural networks; ensembling does not help mispredictions for inputs predicted with a high probability, i.e. the output from the softmax. This finding motivates us to develop a new technique called adaptive ensemble prediction, which achieves the benefits of ensembling with much smaller additional execution costs. Hence, we calculate the confidence level of the prediction for each input from the probabilities of the local predictions during the ensembling computation. If the prediction for an input reaches a high enough probability on the basis of the confidence level, we stop ensembling for this input to avoid wasting computation power. We evaluated the adaptive ensembling by using various datasets and showed that it reduces the computation cost significantly while achieving similar accuracy to the naive ensembling. We also showed that our statistically rigorous confidence-level-based termination condition reduces the burden of the task-dependent parameter tuning compared to the naive termination based on the pre-defined threshold in addition to yielding a better accuracy with the same cost.

The authors propose and evaluate an adaptive ensembling threshold using estimated confidence intervals from the t-distribution, rather than a static confidence level threshold. They show it can provide significant improvements in accuracy at the same cost as a naive threshold.
This paper has a nice simple idea at its core, but I don't think it's fully developed. There's a few major conceptual issues going on:
- The authors propose equation (3) as a stopping criterion because "computing CIs for all labels is costly." I don't see how this is true in any sense. The CI computation is literally just averages of a few numbers, which should be way less than the massive matrix multiplies needed to *generate* those numbers in the neural network. Computing pair-wise comparisons naively in O(n^2) time could potentially blow up if the number of output labels is massive, but then you should still be able to keep some running statistics to avoid having to do a quadratic number of comparisons (e.g. the threshold is just the highest bound of any CI you encounter, so you keep track of both the max predicted confidence and max CI so far...then you have your answer in O(n) time.) I think the real issue is that the authors state that the confidence interval computation code is written in Python. That is a huge knock against this paper: When writing a paper about inference time, it's just due diligence to do the most basic inference time optimizations (such as implementing an operation which should be effectively free in a C++ plugin.)
- So by using (3) instead of the original proposed CI comparison that motivated this approach, the authors require that the predicted probability be greater than 1/2 + the CI at the given alpha level. This means that for problems with very large output spaces, getting enough probability mass to get over that 1/2 absolute threshold is potentially going to require a minimum number of evaluations and put a cap on the efficiency gain. This is what we see in Figure 3: for the few points evaluated, when the output space is large (ILSVRC 2012) there is no effective difference between the proposed method and a static threshold of 70%, indicating that the CI of 90% is roughly working out to be the 50% minimum + ~20% threshold from the CI.
- Thus the experiments in this paper don't really add much value in understanding the benefits of this approach as currently written. For due diligence, there should be the following:
1. Show the distribution of computing thresholds from the CI.  Then compute, for a CI of 0.8, 0.9, etc., what is the effective threshold on average? Then for every *average threshold* from the CI method, apply that as a static threshold. Then you will get exactly the delta of your method over the static threshold method.
2. Do the same, but using the pairwise CI comparison method.
3. The same again, but now show how effective this is as a function of the size of the output label space.  E.g. add these numbers to Table 1 and Table 2 (for every "our adaptive ensemble", put the equivalent static threshold.)
4. Implement the CI computation efficiently if you are going to report actual runtimes. Note that for a paper like this, I don't think the runtimes are as important as the # of evaluations in the ensemble, so this is less important.
- With the above experiments I think this would be a good paper.


Published as a conference paper at ICLR 2018 FIX YOUR CLASSIFIER: THE MARGINAL VALUE OF TRAINING THE LAST WEIGHT LAYER
Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.

Revised Review:
The authors have largely addressed my concerns with the revised manuscript. I still have some doubts about the C > N setting (the new settings of C / N of 4 and 2 aren't C >> N, and the associated results aren't detailed clearly in the paper), but I think the paper warrants acceptance.
Original Review:
The paper proposes fixing the classification layers of neural networks, replacing the traditional learned affine transformation with a fixed (e.g., Hadamard) matrix. This is motivated by the observation that classification layers frequently constitute a non-trivial fraction of a network's overall parameter count, compute requirements, and memory usage, and by the observation that removal of pre-classification fully-connected layers has often been found to have minimal impact on performance. Experiments are performed on a range of datasets and network architectures, in both image classification and NLP settings.
First, I'd like to note that the empirical component of this paper is strong: I was impressed by the breadth of architectures and settings covered, and the experiments left me reasonably convinced that the classification layer can often be fixed, at least for image classification tasks, without significant loss of accuracy.
I have two general concerns. For one, removing the fully connected classification layer is not a novel idea; All Convolutional Networks (https://arxiv.org/abs/1412.6806) reported excellent results without an additional fully connected affine transform (just a global average pooling after the last convolutional layer). I think it would be worth at least referencing/discussing differences with this and other all-convolutional architectures. Including a fixed Hadamard matrix for the classification layer is I believe new (although related to an existing literature on using structured matrices in neural networks).
However, I have doubts about the ability of the approach to scale to problems with a larger number of classes, which arguably is a primary motivation of the paper ("parameters ... grow linearly with the number of classes"). Specifically, the idea of using a fixed N x C matrix with C orthogonal columns (such as Hadamard) is only possible when N > C. This is a critical point: in the N > C regime, a final hidden representation with N dimensions can be chosen to achieve *any* C-dimensional output, regardless of the projection matrix used (so long as it is full rank). This makes it seem fairly reasonable to me that the network can (at least approximately, and complicated by the ReLU nonlinearities) fold the "desired" classification layer into the previous layer, especially with a learned scaling and bias term. In fact it's not clear to me that the fixed classification layer accomplishes anything here, beyond projecting from N -> C (i.e., if N = C, I'd guess it could be removed entirely similar to all convolutional nets, as long as the learned scaling and bias were retained).
On the other hand, when C > N, it is not possible to have mutually orthogonal columns, and in general the output is constrained to lie in an N-dimensional subspace of the overall C-dimensional output space. Picking somewhat randomly a *fixed* N-dimensional subspace seems like a bad idea when N << C, since it is unlikely to select a subspace in which it is possible to adequately capture correlations between the different classes. This makes the proposed technique much less appealing for precisely the family of problems where it would be most effective in reducing compute/memory requirements. It also provides (in my view) a clearer explanation for the failure of the approach in the NLP setting. These issues were not discussed anywhere in the text as far as I can tell, and I think it's necessary to at least acknowledge that mutually orthogonal columns can't be chosen when C > N in section 2.2 (and probably include a longer discussion on the probable implications).
Overall, I think the paper provides a useful observation that clearly isn't common knowledge, since classification layers persist in many popular recent architectures. But the notion of fixing or removing the classification layer isn't particularly novel, and I don't believe the proposed technique would scale well to settings with many classes. As is I think the paper falls slightly short.


UNDERSTANDING SHORT-HORIZON BIAS IN STOCHASTIC META-OPTIMIZATION
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled. But because the training procedure must be unrolled thousands of times, the metaobjective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training. We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias. We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons. We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area. We believe short-horizon bias is a fundamental problem that needs to be addressed if metaoptimization is to scale to practical neural net training regimes.

This paper studies the issue of truncated backpropagation for meta-optimization. Backpropagation through an optimization process requires unrolling the optimization, which due to computational and memory constraints, is typically restricted or truncated to a smaller number of unrolled steps than we would like.
This paper highlights this problem as a fundamental issue limiting meta-optimization approaches. The authors perform a number of experiments on a toy problem (stochastic quadratics) which is amenable to some theoretical analysis as well as a small fully connected network trained on MNIST.
(side note: I was assigned this paper quite late in the review process, and have not carefully gone through the derivations--specifically Theorems 1 and 2).
The paper is generally clear and well written.
Major comments
-------------------------
I was a bit confused why 1000 SGD+mom steps pre-training steps were needed. As far as I can tell, pre-training is not typically done in the other meta-optimization literature? The authors suggest this is needed because "the dynamics of training are different at the very start compared to later stages", which is a bit vague. Perhaps the authors can expand upon  this point?
The conclusion suggests that the difference in greedy vs. fully optimized schedule is due to the curvature (poor scaling) of the objective--but Fig 2. and earlier discussion talked about the noise in the objective as introducing the bias (e.g. from earlier in the paper, "The noise in the problem adds uncertainty to the objective, resulting in failures of greedy schedule"). Which is the real issue, noise or curvature? Would running the problem on quadratics with different condition numbers be insightful?
Minor comments
-------------------------
The stochastic gradient equation in Sec 2.2.2 is missing a subscript: "h_i" instead of "h"
It would be nice to include the loss curve for a fixed learning rate and momentum for the noisy quadratic in Figure 2, just to get a sense of how that compares with the greedy and optimized curves.
It looks like there was an upper bound constraint placed on the optimized learning rate in Figure 2--is that correct? I couldn't find a mention of the constraint in the paper. (the optimized learning rate remains at 0.2 for the first ~60 steps)?
Figure 2 (and elsewhere): I would change 'optimal' to 'optimized' to distinguish it from an optimal curve that might result from an analytic derivation. 'Optimized' makes it more clear that the curve was obtained using an optimization process.
Figure 2: can you change the line style or thickness so that we can see both the red and blue curves for the deterministic case? I assume the red curve is hiding beneath the blue one--but it would be good to see this explicitly.
Figure 4 is fantastic--it succinctly and clearly demonstrates the problem of truncated unrolls. I would add a note in the caption to make it clear that the SMD trajectories are the red curves, e.g.: "SMD trajectories (red) during meta-optimization of initial effective ...". I would also change the caption to use "meta-training losses" instead of "training losses" (I believe those numbers are for the meta-loss, correct?). Finally, I would add a colorbar to indicate numerical values for the different grayscale values.
Some recent references that warrant a mention in the text:
- both of these learn optimizers using longer numbers of unrolled steps:
Learning gradient descent: better generalization and longer horizons, Lv et al, ICML 2017
Learned optimizers that scale and generalize, Wichrowska et al, ICML 2017
- another application of unrolled optimization:
Unrolled generative adversarial networks, Metz et al, ICLR 2017
In the text discussing Figure 4 (middle of pg. 8) , "which is obtained by using..." should be "which are obtained by using..."
In the conclusion, "optimal for deterministic objective" should be "deterministic objectives"


TCAV: Relative concept importance testing with Linear Concept Activation Vectors
Neural networks commonly offer high utility but remain difficult to interpret. Developing methods to explain their decisions is challenging due to their large size, complex structure, and inscrutable internal representations. This work argues that the language of explanations should be expanded from that of input features (e.g., assigning importance weightings to pixels) to include that of higher-level, humanfriendly concepts. For example, an understandable explanation of why an image classifier outputs the label "zebra" would ideally relate to concepts such as "stripes" rather than a set of particular pixel values. This paper introduces the "concept activation vector" (CAV) which allows quantitative analysis of a concept's relative importance to classification, with a user-provided set of input data examples defining the concept. CAVs may be easily used by non-experts, who need only provide examples, and with CAVs the high-dimensional structure of neural networks turns into an aid to interpretation, rather than an obstacle. Using the domain of image classification as a testing ground, we describe how CAVs may be used to test hypotheses about classifiers and also generate insights into the deficiencies and correlations in training data. CAVs also provide us a directed approach to choose the combinations of neurons to visualize with the DeepDream technique, which traditionally has chosen neurons or linear combinations of neurons at random to visualize.

Summary
---
This paper proposes the use of Concept Activation Vectors (CAVs) for interpreting deep models. It shows how concept activation vectors can be used to provide explanations where the user provides a concept (e.g., red) as a set of training examples and then the method provides explanations like "If there were more red in this image then the model would be more likely to classify it as a fire truck."
Four criteria are enumerated for evaluating interpretability methods:
1. accessibility: ML background should not be required to interpret a model
2. customization: Explanations should be generated w.r.t. user-chosen concepts
3. plug-in readiness: Should be no need to re-train/modify the model under study
4. quantification: Explanations should be quantitative and testable
A Concept Activation Vector is simply the weight vector of a linear classifier trained on some examples (100-500) of a user-provided concept of interest using features extracted from an intermediate network layer. These vectors can be trained in two ways:
1. 1-vs-all: The user provides positive examples of a concept and all other existing training data is treated as negatives
2. 1-vs-1: The user provides sets of positive and negative examples, allowing the negative examples to be targeted to one category
Once a CAV is obtained it is used in two ways:
First, it provides further verification that higher level concepts tend to be "disentangled" in deeper network layers while low level concepts are "disentangled" earlier in the network. This work shows that linear classifier accuracy increases significantly using deeper features for higher level concepts but it only increases marginally (or even decreases) when modeling lower level concepts.
Second, and this is the main point of the paper, relative importance of concepts w.r.t. a particular task can be evaluated. Suppose an image (e.g., of a zebra) produces a feature vector f_l at layer l and v_l is a concept vector learned to classify the presence of stripes from layer l features. Then the probability the model assigns to the zebra class can be evaluated using features f_l and then f_l + v^c_l. If the latter probability is greater then adding stripes will increase the model's confidence in the zebra class. Furthermore, the method goes on to measure how often stripes increase zebra confidence across all images. Rather than explaining the network's decision for a particular image, this average metric measures the global importance of the stripes concept for zebra. The paper reports examples of the relative importance of certain concepts with respect to others in figure 5.
Pros
---
The paper proposes a simple and novel idea which could have a major impact on how deep networks are explained. At a high level the novelty comes from replacing the gradient (or something similar) used in saliency methods with a directional derivative. Users can align the direction to any concept they find relevant, so the concept space used to explain a prediction is no longer fixed a-priori (e.g. to pixels in the input space). It can adapt to user suspicions and expectations.
Cons
---
Concerns about story/presentation:
* The second use of CAVs, to test relative importance of concepts, is basically an improved saliency method. It's advantages over other saliency methods are stated clearly in 2.1, but it should not be portrayed as fundamentally different.
The two quantities in eq. 1 can be thought of in terms of directional derivatives. To compute I_w^up start by computing a finite differences approximation of directional derivative of the linear classifier probability p_k(y) with respect to layer l features in the direction of the CAV v_C^l. Call this quantity g_i (for the ith example). Then I_w^up is the average of 1(g_i > 0) over all examples. I think the notion of relative importance used here is basically the idea of a directional derivative.
This doesn't change the contribution of the paper but it should be mentioned and section 2.1 should be changed so it doesn't suggest this method is fundamentally different than saliency methods in terms of criteria 4.
* Evaluation and Desiderata 4: The fourth criteria for interpretability laid out by the paper says an explanation should be quantitative and testable. I'm not sure exactly what this is supposed to mean. I see two ways to interpret the quantitative criterion.
One way to interpret the "quantifiability" criterion is to say that it requires explanations to be presented as numeric values. But most methods do this.  In particular, saliency methods report results in terms of pixel brightness (that is a numeric quantity) even though humans may not know how to interpret that correctly. I do not think this is what was intended, so my second option is to say that the criterion requires an explanation be judged good or bad according to some quantitative metric. But this paper provides no such metric. The explanations in figure 5 are not presented as good or bad according to any metric.
While it is significant that the method meets the first 3 criteria, these do not establish the fidelity of the method. Do humans generalize these explanations to valid inferences about model behavior? Maybe consider some evaluation options from section 3 of Doshi-Velez and Kim 2017 (cited in the paper).
* Section 4.1.1: "This experiment does not yet show that these concept activation vectors align with the concepts that makes sense semantically to humans."
Isn't test set accuracy a better measure of alignment with the human concept than the visualizations? Given a choice between a concept vector which produced good test accuracy and poor visualizations and another concept vector which produced poor test accuracy and good visualizations I would think the one with good test accuracy is better aligned to the human concept. I would still prefer a concept vector which satisfies both.
* Contrary to the description in section 2.2, I think DeepDream optimizes a natural image (non-random initialization) rather than starting from a random image. It looks like these visualization start from a random initialization. Which method is used? Maybe cite this paper, which gives a nice overview: "Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks" by Nguyen et. al. in the Visualization for Deep Learning workshop at ICML 2016
* In section 4.1.3 I'm not quite sure what the point is. Please state it more clearly. Is the context class the same as the negative set used to train the classifier? Why should it be different/harder to sort corgi examples according to a concept vector as opposed to sorting all examples according to a concept vector? This seems like a useful way of testing to be sure CAV's represent human concepts, but I'm not sure what context concepts like striped/CEO provide.
* Relative vs absolute importance and user choice: Section 4.2 claims that figure 5 shows that a CAV "captures an important aspect of the prediction." I would be a bit more careful about the distinction between relative and absolute here. If red makes images more probably fire trucks then it doesn't necessarily mean that red is important for the fire truck concept in an absolute sense. Can we be sure that there aren't other concepts which more dramatically affect outputs? What if a user makes a mistake and only requests explanations with respect to concepts that are irrelevant to the class being explained? Do we need to instruct users on how to best interpret the explanation?
* How practical is this method? Is it a significant burden for users to provide 100-500 images per concept? Are the top 100 or so images from a search engine good enough to specify a CAV?
Minor missing experimental settings and details:
* Section 3 talks about a CAV defined with respect to a non-generic set D of negative examples. Is this setting ever used in the experiments or is the negative set always the same? How does specifying a narrow set of negatives change the CAV for concept C?
* I assume the linear classifier is a logistic regressor, but this is never stated.
* TCAV measures importance/influence as an average over a dataset. This is a validation set, right? For how many of these images are both the user concept and target concept unrelated to the image content (e.g., stripes and zebra for an image of a truck)? When that happens is it reasonable to expect meaningful explanations? They may not be meaningful because the data distribution used to train the CAV probably does not even sparsely cover all concepts in the network's train set. (related to "reference points" in "The (Un)reliability of Saliency Methods" submitted to ICLR18)
* For relative importance testing it would be nice to see a note about the step size selection (1.0) and experiments that show the effect of different step sizes. Hopefully influence is monotonic in step size so that different step sizes do not significantly change the results.
* How large is the typical difference between p_k(y) and p_k(y_w) in eq. 1? If this difference is small then is it meaningful? Are small differences signal or noise?
Final Evaluation
---
I would like to see this idea published, but not in its current form. The method meets a relevant set of criteria that no other method seems to meet, but arguments set forth in the story need some revision and the empirical evaluation needs improvement, especially with respect to model fidelity. I would be happy to change my rating if the above points are addressed.


Deep neural networks (DNN) have shown promising performance in computer vision. In medical imaging, encouraging results have been achieved with deep learning for applications such as segmentation, lesion detection and classification. Nearly all of the deep learning based image analysis methods work on reconstructed images, which are obtained from original acquisitions via solving inverse problems (reconstruction). The reconstruction algorithms are designed for human observers, but not necessarily optimized for DNNs which can often observe features that are incomprehensible for human eyes. Hence, it is desirable to train the DNNs directly from the original data which lie in a different domain with the images. In this paper, we proposed an end-to-end DNN for abnormality detection in medical imaging. To align the acquisition with the annotations made by radiologists in the image domain, a DNN was built as the unrolled version of iterative reconstruction algorithms to map the acquisitions to images, and followed by a 3D convolutional neural network (CNN) to detect the abnormality in the reconstructed images. The two networks were trained jointly in order to optimize the entire DNN for the detection task from the original acquisitions. The DNN was implemented for lung nodule detection in low-dose chest computed tomography (CT), where a numerical simulation was done to generate acquisitions from 1,018 chest CT images with radiologists' annotations. The proposed end-to-end DNN demonstrated better sensitivity and accuracy for the task compared to a two-step approach, in which the reconstruction and detection DNNs were trained separately. A significant reduction of false positive rate on suspicious lesions were observed, which is crucial for the known over-diagnosis in low-dose lung CT imaging. The images reconstructed by the proposed end-to-end network also presented enhanced details in the region of interest.

This paper proposes to jointly model computed tomography reconstruction and lesion detection in the lung, training the mapping from raw sinogram to detection outputs in an end-to-end manner. In practice, such a mapping is computed separately, without regard to the task for wich the data is to be used. Because such a mapping loses information, optimizing such a mapping jointly with the task should preserve more information that is relevant to the task. Thus, using raw medical image data should be useful for lesion detection in CT as well as most other medical image analysis tasks.
Style considerations:
The work is adequately motivated and the writing is generally clear. However, some phrases are awkward and unclear and there are occasional minor grammar errors. It would be useful to ask a native English speaker to polish these up, if possible. Also, there are numerous typos that could nonetheless be easily remedied with some final proofreading. Generally, the work is well articulated with sound structure but needs polish.
A few other minor style points to address:
- "g" is used throughout the paper for two different networks and also to define gradients - if would be more clear if you would choose other letters.
- S3.3, p. 7 : reusing term "iteration"; clarify
- fig 10: label the columns in the figure, not in the description
- fig 11: label the columns in the figure with iterations
- fig 8 not referenced in text
Questions:
1. Before fine-tuning, were the reconstruction and detection networks trained end-to-end (with both L2 loss and cross-entropy loss) or were they trained separately and then joined during fine-tuning?
(If it is the former and not the latter, please make that more clear in the text. I expect that it was indeed the former; in case that it was not, I would expect fully end-to-end training in the revision.)
2. Please confirm: during the fine-tuning phase of training, did you use only the cross-entropy loss and not the L2 loss?
3a. From equation 3 to equation 4 (on an iteration of reconstruction), the network g() was dropped. It appears to replace the diagonal of a Hessian (of R) which is probably a conditioning term. Have you tried training a g() network? Please discuss the ramifications of removing this term.
3b. Have you tracked the condition number of the Jacobian of f() across iterations? This should be like tracking the condition number of the Hessian of R(x).
4. Please discuss: is it better to replace operations on R() with neural networks rather than to replace R()? Why?
5. On page 5, you write "masks for lung regions were pre-calculated". Were these masks manual segmentations or created with an automated method?
6. Why was detection only targetted on "non-small nodules"? Have you tried detecting small nodules?
7. On page 11, you state: "The tissues in lung had much better contrast in the end-to-end network compared to that in the two-step network". I don't see evidence to support that claim. Could you demonstrate that?
8. On page 12, relating to figure 11, you state:
"Whereas both methods kept similar structural component, the end-to-end method had more focus on the edges and tissues inside lung compared to the two-step method. As observed in figure 11(b), the structures of the lung tissue were much more clearer in the end-to-end networks. This observation indicated that sharper edge and structures were of more importance for the detection network than the noise level in the reconstructed images, which is in accordance with human perceptions when radiologists perform the same task."
However, while these claims appear intuitive and such results may be expected, they are not backed up by figure 11. Looking at the feature map samples in this figure, I could not identify whether they came from different populations. I do not see the evidence for "more focus on the edges and tissues inside lung" for the end-to-end method in fig 11. It is also not obvious whether indeed "the structures of the lung tissue were much more clearer" for the end-to-end method, in fig 11. Can you clarify the evidence in support of these claims?
Other points to address:
1. Please report statistical significance for your results (eg. in fig 5b, in the text, etc.). Also, please include confidence intervals in table 2.
2. Although cross-entropy values, detection metrics were not (except for the ROC curve with false positives and false negatives). Please compute: accuracy, precision, and recall to more clearly evaluate detection performance.
3a. "Abnormality detection" implies the detection of anything that is unusual in the data. The method you present targets a very specific abnormality (lesions). I would suggest changing "abnormality detection" to "lesion detection".
3b. The title should also be updated accordingly. Considering also that the presented work is on a single task (lesion detection) and a single medical imaging modality (CT), the current title appears overly broad. I would suggest changing it from "End-to-End Abnormality Detection in Medical Imaging" -- possibly to something like "End-to-End Computed Tomography for Lesion Detection".
Conclusion:
The motivation of this work is valid and deserves attention. The implementation details for modeling reconstruction are also valuable. It is interesting to see improvement in lesion detection when training end-to-end from raw sinogram data.  However, while lung lesion detection is the only task on which the utility of this method is evaluated, detection improvement appears modest. This work would benefit from additional experimental results or improved analysis and discussion.


Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization. We focus on a simple neural network: RELU network with one hidden layer consisting of two RELU units, and show that all local minimizers are global. This combined with recent work of Lee et al. (2017);  show that gradient descent converges to the global minimizer.

In this paper the authors studied the theoretical properties of manifold descent approaches in a standard regression problem, whose regressor is a simple neural network. Leveraged by two recent results in global optimization, they showed that with a simple two-layer ReLU network with two hidden units, the problem with a standard MSE population loss function does not have spurious local minimum points. Based on the results by Lee et al, which shows that first order methods converge to local minimum solution (instead of saddle points), it can be concluded that the global minima of this problem can be found by any manifold descent techniques, including standard gradient descent methods. In general I found this paper clearly written and technically sound. I also appreciate the effort of developing theoretical results for deep learning, even though the current results are restrictive to very simple NN architectures.
Contribution:
As discussed in the literature review section, apart from previous results that studied the theoretical convergence properties for problems that involves a single hidden unit NN, this paper extends the convergence results to problems that involves NN with two hidden units. The analysis becomes considerably more complicated, and the contribution seems to be novel and significant. I am not sure why did the authors mentioned the work on over-parameterization though. It doesn't seem to be relevant to the results of this paper (because the NN architecture proposed in this paper is rather small).
Comments on the Assumptions:
- Please explain the motivation behind the standard Gaussian assumption of the input vector x.
- Please also provide more motivations regarding the assumption of the orthogonality of weights: w_1^\top w_2=0 (or the acute angle assumption in Section 6).
Without extra justifications, it seems that the theoretical result only holds for an artificial problem setting. While the ReLU activation is very common in NN architecture, without more motivations I am not sure what are the impacts of these results.
General Comment:
The technical section is quite lengthy, and unfortunately I am not available to go over every single detail of the proofs. From the analysis in the main paper, I believe the theoretical contribution is correct and sound. While I appreciate the technical contributions, in order to improve the readability of this paper, it would be great to see more motivations of the problem studied in this paper (even with simple examples). Furthermore, it is important to discuss the technical assumptions on the 1) standard Gaussianity of the input vector, and 2) the orthogonality of the weights (and the acute angle assumption in Section 6) on top of the discussions in Section 8.1, as they are critical to the derivations of the main theorems.


Published as a conference paper at ICLR 2018 TRAINING GENERATIVE ADVERSARIAL NETWORKS VIA PRIMAL-DUAL SUBGRADIENT METHODS: A LAGRANGIAN PERSPECTIVE
We relate the minimax game of generative adversarial networks (GANs) to finding the saddle points of the Lagrangian function for a convex optimization problem, where the discriminator outputs and the distribution of generator outputs play the roles of primal variables and dual variables, respectively. This formulation shows the connection between the standard GAN training process and the primal-dual subgradient methods for convex optimization. The inherent connection does not only provide a theoretical convergence proof for training GANs in the function space, but also inspires a novel objective function for training. The modified objective function forces the distribution of generator outputs to be updated along the direction according to the primal-dual subgradient methods. A toy example shows that the proposed method is able to resolve mode collapse, which in this case cannot be avoided by the standard GAN or Wasserstein GAN. Experiments on both Gaussian mixture synthetic data and real-world image datasets demonstrate the performance of the proposed method on generating diverse samples.

This paper formulates GAN as a Lagrangian of a primal convex constrained optimization problem. They then suggest to modify the updates used in the standard GAN training to be similar to the primal-dual updates typically used by primal-dual subgradient methods.
Technically, the paper is sound. It mostly leverages the existing literature on primal-dual subgradient methods to modify the GAN training procedure. I think this is a nice contribution that does yield to some interesting insights. However I do have some concerns about the way the paper is currently written and I find some claims misleading.
Prior convergence proofs: I think the way the paper is currently written is misleading. The authors quote the paper from Ian Goodfellow: “For GANs, there is no theoretical prediction as to
whether simultaneous gradient descent should converge or not.”. However, the f-GAN paper gave a proof of convergence, see Theorem 2 here: https://arxiv.org/pdf/1606.00709.pdf. A recent NIPS paper by (Nagarajan and Kolter, 2017) also study the convergence properties of simultaneous gradient descent. Another problem is of course the assumptions required for the proof that typically don’t hold in practice (see comment below).
Convex-concave assumption: In practice the GAN objective is optimized over the parameters of the neural network rather than the generative distribution. This typically yields a non-convex non-concave optimization problem. This should be mentioned in the paper and I would like to see a discussion concerning the gap between the theory and the practical algorithm.
Relation to existing regularization techniques: Combining Equations 11 and 13, the second terms acts as a regularizer that minimizes [\lapha f_1(D(x_i))]^2. This looks rather similar to some of the recent regularization techniques such as
Improved Training of Wasserstein GANs, https://arxiv.org/pdf/1704.00028.pdf
Stabilizing Training of Generative Adversarial Networks through Regularization, https://arxiv.org/pdf/1705.09367.pdf
Can the authors comment on this? I think this would also shed some light as to why this approach alleviates the problem of mode collapse.
Curse of dimensionality: Nonparametric density estimators such as the KDE technique used in this paper suffer from the well-known curse of dimensionality. For the synthetic data, the empirical evidence seem to indicate that the technique proposed by the authors does work but I’m not sure the empirical evidence provided for the MNIST and CIFAR-10 datasets is sufficient to judge whether or not the method does help with mode collapse. The inception score fails to capture this property. Could the authors explore other quantitative measure? Have you considered trying your approach on the augmented version of the MNIST dataset used in Metz et al. (2016) and Che et al. (2016)?
Experiments
Typo: Should say “The data distribution is p_d(x) = 1{x=1}”.


Published as a conference paper at ICLR 2018 LEARNING LATENT REPRESENTATIONS IN NEURAL NETWORKS FOR CLUSTERING THROUGH PSEUDO SUPERVISION AND GRAPH-BASED ACTIVITY REGULARIZATION
In this paper, we propose a novel unsupervised clustering approach exploiting the hidden information that is indirectly introduced through a pseudo classification objective. Specifically, we randomly assign a pseudo parent-class label to each observation which is then modified by applying the domain specific transformation associated with the assigned label. Generated pseudo observation-label pairs are subsequently used to train a neural network with Auto-clustering Output Layer (ACOL) that introduces multiple softmax nodes for each pseudo parent-class. Due to the unsupervised objective based on Graph-based Activity Regularization (GAR) terms, softmax duplicates of each parent-class are specialized as the hidden information captured through the help of domain specific transformations is propagated during training. Ultimately we obtain a k-means friendly latent representation. Furthermore, we demonstrate how the chosen transformation type impacts performance and helps propagate the latent information that is useful in revealing unknown clusters. Our results show state-of-the-art performance for unsupervised clustering tasks on MNIST, SVHN and USPS datasets, with the highest accuracies reported to date in the literature.

This paper presents a method for clustering based on latent representations learned from the classification of transformed data after pseudo-labellisation corresponding to applied transformation. Pipeline: -Data are augmented with domain-specific transformations. For instance, in the case of MNIST, rotations with different degrees are applied. All data are then labelled as "original" or "transformed by ...(specific transformation)". -Classification task is performed with a neural network on augmented dataset according to the pseudo-labels. -In parallel of the classification, the neural network also learns the latent representation in an unsupervised fashion. -k-means clustering is performed on the representation space observed in the hidden layer preceding the augmented softmax layer.
Detailed Comments:
(*) Pros
-The method outperforms the state-of-art regarding unsupervised methods for handwritten digits clustering on MNIST.
-Use of ACOL and GAR is interesting, also the idea to make "labeled" data from unlabelled ones by using data augmentation.
(*) Cons
-minor: in the title, I find the expression "unsupervised clustering" uselessly redundant since clustering is by definition unsupervised.
-Choice of datasets: we already obtained very good accuracy for the classification or clustering of handwritten digits. This is not a very challenging task.
And just because something works on MNIST, does not mean it works in general.
What are the performances on more challenging datasets like colored images (CIFAR-10, labelMe, ImageNet, etc.)?
-This is not clear what is novel here since ACOL and GAR already exist. The novelty seems to be in the adaptation to GAR from the semi-supervised to the unsupervised setting with labels indicating if data have been transformed or not.
My main problem  was about the lack of novelty. The authors clarified this point, and it turned out that ACOL and GAR have never published elsewhere except in ArXiv.  The other issue concerned the validation of the approach on databases other than MNIST. The author also addressed this point, and I changed my scores accordingly.


Published as a conference paper at ICLR 2018 SEMANTICALLY DECOMPOSING THE LATENT SPACES OF GENERATIVE ADVERSARIAL NETWORKS
We propose a new algorithm for training generative adversarial networks that jointly learns latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). By fixing the identity portion of the latent codes, we can generate diverse images of the same subject, and by fixing the observation portion, we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce pairs that are photorealistic, distinct, and appear to depict the same individual. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to facilitate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm's ability to generate convincing, identity-matched photographs.

[Overview]
In this paper, the authors proposed a model called SD-GAN, to decompose semantical component of the input in GAN. Specifically, the authors proposed a novel architecture to decompose the identity latent code and non-identity latent code. In this new architecture, the generator is unchanged while the discriminator takes pair data as the input, and output the decision of whether two images are from the same identity or not. By training the whole model with a conventional GAN-training regime, SD-GAN learns to take a part of the input Z as the identity information, and the other part of input Z as the non-identity (or attribute) information. In the experiments, the authors demonstrate that the proposed SD-GAN could generate images preserving the same identity with diverse attributes, such as pose, age, expression, etc. Compared with AC-GAN, the proposed SD-GAN achieved better performance in both automatically evaluation metric (FaceNet) and Human Study. In the appendix, the authors further presented ablated qualitative results in various settings.
[Strengths]
1. This paper proposed a simple but effective generative adversarial network, called SD-GAN, to decompose the input latent code of GAN into separate semantical parts. Specifically, it is mainly instantiated on face images, to decompose the identity part and non-identity part in the latent code. Unlike the previous works such as AC-GAN, SD-GAN exploited a Siamese network to replace the conventional discriminator used in GAN. By this way, SD-GAN could generate images of novel identities, rather than being constrained to those identities used during training. I think this is a very good property. Due to this, SD-GAN consumes much less memory than AC-GAN, when training on a large number of identities.
2. In the experiment section, the authors quantitatively evaluate the generated images based on two methods, one is using a pre-trained FaceNet model to measure the verification accuracy and one is human study. When evaluated based on FaceNet, the proposed SD-GAN achieved higher accuracy and obtained more diverse face images, compared with AC-GAN. In human study, SD-GAN achieved comparable verification accuracy, while higher diversity than AC-GAN. The authors further presented ablated experiments in the Appendix.
[Comments]
This paper presents a novel model to decompose the latent code in a semantic manner. However, I have several questions about the model:
1. Why would SD-GAN not generate images merely have a smaller number of identities or just a few identities? In Algorithm 1, the authors trained the model by sampling one identity vector, which is then concatenated to two observation vectors. In this case, the generator always takes the same identity vectors, and the discriminator is used to distinguish these fake same-identity pair and the real same-identity pair from training data. As such, even if the generator generates the same identity, say mean identity, given different identity vectors, the generated images can still obtain a lower discrimination loss. Without any explicite constraint to enforce the generator to generate different identity with different identity vectors, I am wondering what makes SD-GAN be able to generate diverse identities?
2. Still about the identity diversity. Though the authors showed the identity-matched diversity in the experiments, the diversity across identity on the generated images is not evaluated. The authors should also evaluate this kind of identity. Generally, AC-GAN could generate as many identities as the number of identities in training data. I am curious about whether SD-GAN could generate comparable diverse identity to AC-GAN. One simple way is to evaluate the whole generated image set using Inception Score based on a Pre-trained face identification network; Another way is to directly use the generated images to train a verification model or identification model and evaluate it on real images. Though compared with AC-GAN, SD-GAN achieved better identity verification performance and sample diversity, I suspect the identity diversity is discounted, though SD-GAN has the property of generating novel identities. Furthermore,  the authors should also compare the general quality of generated samples with DC-GAN and BEGAN as well (at least qualitatively), apart from the comparison to AC-GAN on the identity-matched generation.
3. When making the comparison with related work, the authors mentioned that Info-GAN was not able to determine which factors are assigned to each dimension. I think this is not precise. The lack of this property is because there are no data annotations. Given the data annotations, Info-GAN can be easily augmented with such property by sending the real images into the discriminator for classification. Also, there is a typo in the caption of Fig. 10. It looks like each column shares the same identity vector instead of each row.
[Summary]
This paper proposed a new model called SD-GAN to decompose the input latent code of GAN into two separete semantical parts, one for identity and one for observations. Unlike AC-GAN, SD-GAN exploited a Siamese architecture in discriminator. By this way, SD-GAN could not only generate more identity-matched face image pairs but also more diverse samples with the same identity,  compared with AC-GAN. I think this is a good idea for decomposing the semantical parts in the latent code, in the sense that it can imagine new face identities and consumes less memory during training. Overall, I think this is a good paper. However, as I mentioned above, I am still not clear why SD-GAN could generate diverse identities without any constraints to make the model do that. Also, the authors should further evaluate the diversity of identity and compare it with AC-GAN.


In spite of their great success, traditional factorization algorithms typically do not support features (e.g., Matrix Factorization), or their complexity scales quadratically with the number of features (e.g, Factorization Machine). On the other hand, neural methods allow large feature sets, but are often designed for a specific application. We propose novel deep factorization methods that allow efficient and flexible feature representation. For example, we enable describing items with natural language with complexity linear to the vocabulary size-this enables prediction for unseen items and avoids the cold start problem. We show that our architecture can generalize some previously published single-purpose neural architectures. Our experiments suggest improved training times and accuracy compared to shallow methods.

This paper presents a method for matrix factorization using DNNs. The suggestion is to make the factorization machine (eqn 1) deep, by grouping the features meaningfully (eqn 5), extracting nonlinear features from original inputs (deep-in, eqn 8), and adding additional nonlinearity after computing pairwise interactions (deep-out, eqn 7). From the methodology point of view, such extensions are relatively straightforward. As an example, from the experimental results, it seems the grouping of features is done mostly with domain knowledge (e.g., months of year) and not learned automatically. The authors claim the proposed method can circumvent the cold-start problem, and presented some experimental results on recommendation systems with text features.
While the application problems look quite interesting, in my opinion, the paper needs to make the context and contribution clearer. In particular, there is a huge literature in collaborative filtering, and I believe there is by now sufficient work on collaborative filtering with input features (and possibly dealing with the cold-start problem). I think this paper does not connect very well with that literature. When reading it, at times I felt the main purpose of this paper is to solve the application problems presented in experimental results, instead of proposing a general framework. I suggest the authors to demonstrate their method on some well-known datasets (e.g., MovieLens, Netflix), to give the readers an idea if the proposed method is indeed advantageous over more classical methods, or if the success of this paper is mostly due to clever processing of text features using DNNs.
Some detailed comments:
1. eqn 4 does not indicate any rank-r factors.
2. some statements do not seem straightforward/justified to me:
    -- the paper uses the word "inference" several times without definition
    -- "if we were interested in interpreting the parameters, we could constrain w to be non-negative ... ". Is this easy to do, and can the authors demonstrate this in their experiments and show interpretable examples?
    -- "Note that if the dot product is replaced with a neural function, fast inference for cold-start ...".
3. the experimental setup seems quite unusual to me: "since we only observe positive labels, for such tasks in the test set we sample a labels according to the label frequency". This seems very problematic if most of the entries are not observed. Why cannot you use the typical evaluation procedure for collaborative filtering, where you hide some known entries during model training, and evaluate on these entries during test?


We present 3C-GAN: a novel multiple generators structures, that contains one conditional generator that generates a semantic part of an image conditional on its input label, and one context generator generates the rest of an image. Compared to original GAN model, this model has multiple generators and gives control over what its generators should generate. Unlike previous multi-generator models use a subsequent generation process, that one layer is generated given the previous layer, our model uses a process of generating different part of the images together. This way the model contains fewer parameters and the generation speed is faster. Specifically, the model leverages the label information to separate the object from the image correctly. Since the model conditional on the label information does not restrict to generate other parts of an image, we proposed a cost function that encourages the model to generate only the succinct part of an image in terms of label discrimination. We also found an exclusive prior on the mask of the model help separate the object. The experiments on MNIST, SVHN, and CelebA datasets show 3C-GAN can generate different objects with different generators simultaneously, according to the labels given to each generator.

[Overview]
This paper proposed a new generative adversarial network, called 3C-GAN for generating images in a composite manner. In 3C-GAN, the authors exploited two generators, one (G1) is for generating context images, and the other one (G2) is for generating semantic contents. To generate the semantic contents, the authors introduced a conditional GAN scheme, to force the generated images to match the annotations. After generating both parts in parallel, they are combined using alpha blending to compose the final image. This generated image is then sent to the discriminator. The experiments were conducted on three datasets, MNIST, SVHN and MS-CelebA. The authors showed qualitative results on all three datasets, demonstrating that AC-GAN could disentangle the context part from the semantic part in an image, and generate them separately.
[Strenghts]
This paper introduced a layered-wise image generation, which decomposed the image into two separate parts: context part, and semantic part. Corresponding to these two parts are two generators. To ensure this, the authors introduced three strategies:
1. Adding semantic labels: the authors used image semantic labels as the input and then exploited a conditional GAN to enforce one of the generators to generate semantic parts of images. As usual, the label information was added as the input of generator and discriminator as well.
2. Adding label difference cost: the intuition behind this loss is that changing the label condition should merely affect the output of G2. Based on this, outputs of Gc should not change much when flipping the input labels.
3. Adding exclusive prior: the prior is that the masks of context part (m1) and semantic part (m2) should be exclusive to each other. Therefore, the authors added another loss to reduce the sum of component-wise multiplication between m1 and m2.
Decomposing the semantic part from the context part in an image based on a generative model is an interesting problem. However, to my opinion, completing it without any supervision is challenging and meaningless. In this paper, the authors proposed a conditional way to generate images compositionally. It is an interesting extension of previous works, such as Kwak & Zhang (2016) and Yang (2017).
[Weaknesses]
This paper proposed an interesting and intuitive image generation model. However, there are several weaknesses existed:
1. There is no quantitative evaluation and comparisons. From the limited qualitative results shown in Fig.2-10, we can hardly get a comprehensive sense about the model performance. The authors should present some quantitative evaluations in the paper, which are more persuasive than a number of examples. To do that, I suggest the authors exploited evaluation metrics, such as Inception Score to evaluate the overall generation performance. Also, in Yang (2017) the authors proposed adversarial divergence, which is suitable for evaluating the conditional generation. Hence, I suggest the authors use a similar way to evaluate the classification performance of classification model trained on the generated images. This should be a good indicator to show whether the proposed 3C-GAN could generate more realistic images which facilitate the training of a classifier.
2. The authors should try more complicated datasets, like CIFAR-10. Recently, CIFAR-10 has become a popular dataset as a testbed for evaluating various GANs. It is easy to train since its low resolution, but also means a lot since it a relative complicated scene. I would suggest the authors also run the experiments on CIFAR-10.
3. The authors did not perform any ablation study. Apart from several generation results based on 3C-GAN, iIcould not found any generation results from ablated models. As such, I can hardly get a sense of the effects of different losses and know about the relative performance in the whole GAN spectrum. I strongly suggest the authors add some ablation studies. The authors should at least compare with one-layer conditional GAN.
4. The proposed model merely showed two-layer generation results. There might be two reasons: one is that it is hard to extend it to more layer generation as I know, and the other one reason is the inflexible formulation to compose an image in 2.2.1 and formula (6). The authors should try some datasets like MNIST-TWO in Yang (2017) for demonstration.
5. Please show f1, m1, f2, m2 separately, instead of showing the blending results in Fig3, Fig4, Fig6, Fig7, Fig9, and Fig10. I would like to see what kind of context image and foreground image 3C-GAN has generated so that I can compare it with previous works like Kwak & Zhang (2016) and Yang (2017).
6. I did not understand very well the label difference loss in (5). Reducing the different between G_c(z_u, z_v, z_l) and G_c(z_u, z_v, z_l^f) seems not be able to force G1 and G2 to generate different parts of an image. G2 takes all the duty  can still obtain a lower L_ld. From my point of view, the loss should be added to G1 to make G1 less prone to the variation of label information.
7. Minor typos and textual errors. In Fig.1, should the right generator be G2 rather than G1? In 2.1.3 and 2.2.1, please add numbers to the equations.
[Summary]
This paper proposed an interesting way of generating images, called 3C-GAN. It generates images in a layer-wise manner. To separate the context and semantic part in an image, the authors introduced several new techniques to enforce the generators in the model undertake different duties. In the experiments, the authors showed qualitative results on three datasets, MNIST, SVHN and CelebA. However, as I pointed out above, the paper missed quantitative evaluation and comparison, and ablation study. Taking all these into account, I think this paper still needs more works to make it solid and comprehensive before being accepted.


CRITICAL PERCOLATION AS A FRAMEWORK TO ANALYZE THE TRAINING OF DEEP NETWORKS
In this paper we approach two relevant deep learning topics: i) tackling of graph structured input data and ii) a better understanding and analysis of deep networks and related learning algorithms. With this in mind we focus on the topological classification of reachability in a particular subset of planar graphs (Mazes). Doing so, we are able to model the topology of data while staying in Euclidean space, thus allowing its processing with standard CNN architectures. We suggest a suitable architecture for this problem and show that it can express a perfect solution to the classification task. The shape of the cost function around this solution is also derived and, remarkably, does not depend on the size of the maze in the large maze limit. Responsible for this behavior are rare events in the dataset which strongly regulate the shape of the cost function near this global minimum. We further identify an obstacle to learning in the form of poorly performing local minima in which the network chooses to ignore some of the inputs. We further support our claims with training experiments and numerical analysis of the cost function on networks with up to 128 layers.

This paper thoroughly analyzes an algorithmic task (determining if two points in a maze are connected, which requires BFS to solve) by constructing an explicit ConvNet solution and analytically deriving properties of the loss surface around this analytical solution. They show that their analytical solution implements a form of BFS algorithm, characterize the probability of introducing "bugs" in the algorithm as the weights move away from the optimal solution, and how this influences the error surface for different depths. This analysis is conducted by drawing on results from the field of critical percolation in physics.
Overall, I think this is a good paper and its core contribution is definitely valuable: it provides a novel analysis of an algorithmic task which sheds light on how and when the network fails to learn the algorithm, and in particular the role which initialization plays. The analysis is very thorough and the methods described may find use in analyzing other tasks. In particular, this could be a first step towards better understanding the optimization landscape of memory-augmented neural networks (Memory Networks, Neural Turing Machines, etc) which try to learn reasoning tasks or algorithms. It is well-known that these are sensitive to initialization and often require running the optimizer with multiple random seeds and picking the best one. This work actually explains the role of initialization for learning BFS and how certain types of initialization lead to poor solutions. I am curious if a similar analysis could be applied to methods evaluated on the bAbI question-answering tasks (which can be represented as graphs, like the maze task) and possibly yield better initialization or optimization schemes that would remove the need for multiple random seeds.
With that being said, there is some work that needs to be done to make the paper clearer. In particular, many parts are quite technical and may not be accessible to a broader machine learning audience. It would be good if the authors spent more time developing intuition (through visualization for example) and move some of the more technical proofs to the appendix. Specifically:
- I think Figure 3 in the appendix should be moved to the main text, to help understand the behavior of the analytical solution.
- Top of page 5, when you describe the checkerboard BFS: please include a visualization somewhere, it could be in the Appendix.
- Section 6: there is lots of math here, but the main results don't obviously stand out. I would suggest highlighting equations 2 and 4 in some way (for example, proposition/lemma + proof), so that the casual reader can quickly see what the main results are. Interested readers can then work through the math if they want to. Also, some plots/visualizations of the loss surface given in Equations 4 and 5 would be very helpful.
Also, although I found their work to be interesting after finishing the paper, I was initially confused by how the authors frame their work and where the paper was heading. They claim their contribution is in the analysis of loss surfaces (true) and neural nets applied to graph-structured inputs. This second part was confusing - although the maze can be viewed as a graph, many other works apply ConvNets to maze environments [1, 2, 3], and their work has little relation to other work on graph CNNs. Here the assumptions of locality and stationarity underlying CNNs are sensible and I don't think the first paragraph in Section 3 justifying the use of the CNN on the maze environment is necessary. However, I think it would make much more sense to mention how their work relates to other neural network architectures which learn algorithms (such as the Neural Turing Machine and variants) or reasoning tasks more generally (for example, memory-augmented networks applied to the bAbI tasks).
There are lots of small typos, please fix them. Here are a few:
- "For L=16, batch size of 20, ...": not a complete sentence.
- Right before 6.1.1: "when the these such" -> "when such"
- Top of page 8: "it also have a" -> "it also has a", "when encountering larger dataset" -> "...datasets"
-  First sentence of 6.2: "we turn to the discuss a second" -> "we turn to the discussion of a second"
- etc.
Quality: High
Clarity: medium-low
Originality: high
Significance: medium-high
References:
[1] https://arxiv.org/pdf/1602.02867.pdf
[2] https://arxiv.org/pdf/1612.08810.pdf
[3] https://arxiv.org/pdf/1707.03497.pdf


The tasks that an agent will need to solve often aren't known during training. However, if the agent knows which properties of the environment are important, then after learning how its actions affect those properties the agent may be able to use this knowledge to solve complex tasks without training specifically for them. Towards this end, we consider a setup in which an environment is augmented with a set of user defined attributes that parameterize the features of interest. We propose a method that learns a policy for transitioning between "nearby" sets of attributes, and maintains a graph of possible transitions. Given a task at test time that can be expressed in terms of a target set of attributes, and a current state, our model infers the attributes of the current state and searches over paths through attribute space to get a high level plan, and then uses its low level policy to execute the plan. We show in grid-world games and 3D block stacking that our model is able to generalize to longer, more complex tasks at test time even when it only sees short, simple tasks at train time.

Summary: This paper proposes a method for planning which involves learning to detect high-level subgoals (called "attributes"), learning a transition model between subgoals, and then learning a policy for the low-level transitions between subgoals. The high-level task plan is not learned, but is computed using Dijkstra's algorithm. The benefit of this method (called the "Attribute Planner", or AP) is that it is able to generalize to tasks requiring multi-step plans after only training on tasks requiring single-step plans. The AP is compared against standard A3C baselines across a series of experiments in three different domains, showing impressive performance and demonstrating its generalization capability.
Pros:
- Impressive generalization results on multi-step planning problems.
- Nice combination of model-based planning for the high-level task plan with model-free RL for the low-level actions.
Cons:
- Attributes are handcrafted and pre-specified rather than being learned.
- Rather than learning an actual parameterized high-level transition model, a graph is built up out of experience, which requires a large sample complexity.
- No comparison to other hierarchical RL approaches.
Quality and Clarity:
This is a great paper. It is extremely well written and clear, includes a very thorough literature review (though it should probably also discuss [1]), takes a sensible approach to combining high- and low-level planning, and demonstrates significant improvements over A3C baselines when generalizing to more complex task plans. The experiments and domains seem reasonable (though the block-stacking domain would be more interesting if the action and state spaces weren't discrete) and the analysis is thorough.
While the paper in general is written very clearly, it would be helpful to the reader to include an algorithm for the AP.
Originality and Significance:
I am not an expert in hierarchical RL, but my understanding is that typically hierarchical RL approaches use high-level goals to make the task easier to learn in the first place, such as in tasks with long planning horizons (e.g. Montezuma's Revenge). The present work differs from this in that, as they state, "the goal of the model is to be able to generalize to testing on complex tasks from training on simpler tasks" (pg. 5). Most work I have seen does not explicitly test for this generalization capability, but this paper points out that it is important and worthwhile to test for.
It is difficult to say how much of an improvement this paper is on top of other related hierarchical RL works as there are no comparisons made. I think it would be worthwhile to include a comparison to other hierarchical RL architectures (such as [1] or [2]), as I expect they would perform better than the A3C baselines. I suspect that the AP would still have better generalization capabilities, but it is hard to know without seeing the results. That said, I still think that the contribution of the present paper stands on its own.
[1] Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017). FeUdal Networks for Hierarchical Reinforcement Learning. Retrieved from http://arxiv.org/abs/1703.01161
[2] Kulkarni, T. D., Narasimhan, K. R., Saeedi, A., & Tenenbaum, J. B. (2016). Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. Advances in Neural Information Processing Systems.


Graph Convolutional Networks (GCNs) are a recently proposed architecture which has had success in semi-supervised learning on graph-structured data. At the same time, unsupervised learning of graph embeddings has benefited from the information contained in random walks. In this paper we propose a model, Network of GCNs (N-GCN), which marries these two lines of work. At its core, N-GCN trains multiple instances of GCNs over node pairs discovered at different distances in random walks, and learns a combination of the instance outputs which optimizes the classification objective. Our experiments show that our proposed N-GCN model achieves state-of-the-art performance on all of the challenging node classification tasks we consider: Cora, Citeseer, Pubmed, and PPI. In addition, our proposed method has other desirable properties, including generalization to recently proposed semi-supervised learning methods such as GraphSAGE, allowing us to propose N-SAGE, and resilience to adversarial input perturbations.

In this work a new network of GCNs is proposed. Different GCNs utilize different powers of the transition matrix to capture varying neighborhoods in a graph. As an aggregation mechanism of the GCN modules two approaches are considered: a fully connected layer on top of stacked features and attention mechanism that uses a scalar weight per GCN. The later allows for better interpretability of the effects of varying degree of neighborhoods in a graph.
Proposed approach, as authors noted themselves, is quite similar to DCNN (Atwood and Towsley, 2016) and becomes equivalent if the combined GCNs have one layer each. While comparison to vanilla GCN is quite extensive, there is no comparison to DCNN at all. I would be curious to see at least portion of the experiments of the DCNN paper with the proposed approach, where the importance of number of GCN layers is addressed. DCNN did well on Cora and Pubmed when more training samples were used. It also was tested on graph classification datasets, but the results were not as good for some of the datasets. I think that comparison to DCNN is important to justify the importance of using multilayer GCN modules.
Some questions and concerns:
- I could not quite figure out how many layers did each GCN have in the experiments and how impactful is this parameter
- Why is it necessary to replicate GCNs for each of the transition matrix powers? In section 4.3 it is mentioned that replication factors r = 1 and r = 4 were used, but it is not clear from Table 2 what are the results for respective r.
- Early stopping implementation seems a bit too intense. "We invoke many runs over all datasets" - how many? Mean and standard deviation are reported for top 3 performers, which is not enough to get a sense of standard deviation and mean. Kipf and Welling (2017) report results over 100 runs without selecting top performers if I understood correctly their setup. Could you please report mean and standard deviation of all the runs? Given relatively small performance improvement (comparatively to GCN), more than 3 (selected) runs are needed for comparison.
- I liked the attention idea and its interpretation in Fig. 2. Could you please add the error bars for the attention weights. It is interesting to see them shifting towards higher powers of the transition matrix, but also it is important to know if this phenomena is statistically significant.
- Following up on the previous item - did you try not including self connections when computing transition matrix powers? This way the effect of different degrees of neighborhoods in a graph could be understood better. When self-connections are present, each subsequent transition matrix power contains neighborhoods of lower degrees and interpretation becomes not as apparent.
Minor comments:
- Understanding of this paper quite heavily relies on the reader knowing Kipf and Welling (2017) paper. Particularly, the comment about approximations derived by Kipf and Welling (2017) in Section 3.3 and how directed graph was converted to undirected (Section 4.1) require a bit more details.
- I am not quite sure why Section 2.3 is needed. Connection to graph embeddings is not given much attention in the paper later on (except t-SNE picture).
- Typo in Fig. 1 caption - right and left are mixed up.
- Typo in footnote on page 3.


HIERARCHICAL REPRESENTATIONS FOR EFFICIENT ARCHITECTURE SEARCH
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.

The fundamental contribution of the article is the explicit use of compositionality in the definition of the search space. Instead of merely defining an architecture as a Directed Acyclic Graph (DAG), with nodes corresponding to feature maps and edges to primitive operations, the approach in this paper introduces a hierarchy of architectures of this form. Each level of the hierarchy utilises the existing architectures in the preceding level as candidate operations to be applied in the edges of the DAG. As a result, this would allow the evolutionary search algorithm to design modules which might be then reused in different edges of the DAG corresponding to the final architecture, which is located at the top level in the hierarchy.
Manually designing novel neural architectures is a laborious, time-consuming process. Therefore, exploring new approaches to automatise this task is a problem of great relevance for the field.
Overall, the paper is well-written, clear in its exposition and technically sound. While some hyperparameter and design choices could perhaps have been justified in greater detail, the paper is mostly self-contained and provides enough information to be reproducible.
The fundamental contribution of this article, when put into the context of the many recent publications on the topic of automatic neural architecture search, is the introduction of a hierarchy of architectures as a way to build the search space. Compared to existing work, this approach should emphasise modularity, making it easier for the evolutionary search algorithm to discover architectures that extensively reuse simpler blocks as part of the model. Exploiting compositionality in model design is not novel per se (e.g. [1,2]), but it is to the best of my knowledge the first explicit application of this idea in neural architecture search.
Nevertheless, while the idea behind the proposed approach is definitely interesting, I believe that the experimental results do not provide sufficiently compelling evidence that the resulting method substantially outperforms the non-hierarchical, flat representation of architectures used in other publications. In particular, the results highlighted in Figure 3 and Table 1 seem to indicate that the difference in performance between both paradigms is rather small. Moreover, the performance gap between the flat and hierarchical representations of the search space, as reported in Table 1, remains smaller than the performance gap between the best performing of the approaches proposed in this article and NASNet-A (Zoph et al., 2017), as reported in Tables 2 and 3.
Another concern I have is regarding the definition of the mutation operators in Section 3.1. While not explicitly stated, I assume that all sampling steps are performed uniformly at random (otherwise please clarify it). If that was indeed the case, there is a systematic asymmetry between the probability to add and remove an edge, making the former considerably more likely. This could bias the architectures towards fully-connected DAGs, as indeed seems to occur based on the motifs reported in Appendix A.
Finally, while the main motivation behind neural architecture search is to automatise the design of new models, the approach here presented introduces a non-negligible number of hyperparameters that could potentially have a considerable impact and need to be selected somehow. This includes, for instance, the number of levels in the hierarchy (L), the number of motifs at each level in the hierarchy (M_l), the number of nodes in each graph at each level in the hierarchy (| G^{(l)} |), as well as the set of primitive operations. I believe the paper would be substantially strengthened if the authors explored how robust the resulting approach is with respect to perturbations of these hyperparameters, and/or provided users with a principled approach to select reasonable values.
References:
[1] Grosse, Roger, et al. "Exploiting compositionality to explore a large space of model structures." UAI (2012).
[2] Duvenaud, David, et al. "Structure discovery in nonparametric regression through compositional kernel search." ICML (2013).


This paper proposes a new model for document embedding. Existing approaches either require complex inference or use recurrent neural networks that are difficult to parallelize. We take a different route and use recent advances in language modeling to develop a convolutional neural network embedding model. This allows us to train deeper architectures that are fully parallelizable. Stacking layers together increases the receptive filed allowing each successive layer to model increasingly longer range semantic dependences within the document. Empirically we demonstrate superior results on two publicly available benchmarks. Full code will be released with the final version of this paper.

This paper proposes a new model for the general task of inducing document representations (embeddings). The approach uses a CNN architecture, distinguishing it from the majority of prior efforts on this problem, which have tended to use RNNs. This affords obvious computational advantages, as training may be parallelized.
Overall, the model presented is relatively simple (a good thing, in my view) and it indeed seems fast. I can thus see potential practical uses of this CNN based approach to document embedding in future work on language tasks. The training strategy, which entails selecting documents and then indexes within them stochastically, is also neat. Furthermore, the work is presented relatively clearly. That said, my main concerns regarding this paper are that: (1) there's not much new here, and, (2) the experimental setup may be flawed, in that it would seem model hyperparams were tuned for the proposed approach but not for the baselines; I elaborate on these concerns below.
Specific comments:
---
- It's hard to tease out exactly what's new here: the various elements used are all well known. But perhaps there is merit in putting the specific pieces together. Essentially, the novelty is using a CNN rather than an RNN to induce document embeddings.
- In Section 4.1, the authors write that they report results for their after running "parameter sweeps ..." -- I presume that these were performed on a validation set, but the authors should say so. In any case, a very potential weakness here: were analagous parameter sweeps for this dataset performed for the baseline models? It would seem not, as the authors write "the IMDB training data using the default hyper-parameters" for skip-thought. Surely it is unfair comparison if one model has been tuned to a given dataset while others use only the default hyper-parameters?
- Many important questions were left unaddressed in the experiments. For example, does one really need to use the gating mechanism borrowed from the Dauphin et al. paper? What happens if not? How big of an effect does the stochastic sampling of document indices have on the learned embeddings? Does the specific underlying CNN architecture affect results, and how much? None of these questions are explored.
- I was left a bit confused regarding how the v_{1:i-1} embedding is actually estimated; I think the details here are insufficient in the current presentation. The authors write that this is a "function of all words up to w_{i-1}". This would seem to imply that at test time, prediction is not in fact parallelizable, no? Yet this seems to be one of the main arguments the authors make in favor of the model (in contrast to RNN based methods). In fact, I think the authors are proposing using the (aggregated) filter activation vectors (h^l(x)) in eq. 5, but for some reason this is not made explicit.
Minor comments:
- In Eq. 4, should the product be element-wise to realize the desired gating (as per the Dauhpin paper)? This should be made explicit in the notation.
- On the bottom of page 3, the authors claim "Expanding the prediction to multiple words makes the problem more difficult since the only way to achieve that is by 'understanding' the preceding sequence." This claim should either by made more precise or removed. It is not clear exactly what is meant here, nor what evidence supports it.
- Commas are missing in a few. For example on page 2, probably want a comma after "in parallel" (before "significantly"); also after "parallelize" above "Approach".
- Page 4: "In contrast, our model addresses only requires" --> drop the "addresses".