RETHINKING THE VALUE OF NETWORK PRUNING
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

This paper reinvestigate several recent works on network pruning and find that the common belief about the necessity to train a large network before pruning may not hold. The authors find that training the pruned model from scratch can achieve similar, if not better, performance given enough time of training. Based on these observations, the author conclude that training a larger model followed by pruning is not necessary for obtaining an efficient model with similar performance. In other words, the pruned architecture is more important than the weights inherited from the large model. It reminds researchers to perform stronger baselines before showing complex pruning methods.
The paper is well organized and written. It re-evaluate the recent progresses made on this topic. Instead of comparing approaches by simply using the numbers from previous paper, the authors perform extensive experiments to verify whether training the pruned network from scratch would work. The results are very interesting, it suggests the researchers to tune the baseline “hardly” and stick to simple approach. However, here are some places that I have concerns with:
1. The two “common beliefs” actually state one thing, that is the weights of a pre-trained larger model can potentially help optimization for a smaller model.
2. I don’t quite agree with that “training” is the first step of a pruning pipeline as illustrated in Figure 1.  Actually the motivation or the common assumption for pruning is that there are already existing trained models (training is already finished) with good performance. If a trained model does not even exist, then one can certainly train various thin/smaller model from scratch as before, this is still a trial and error process.
3. “The value of pruning”. The goal of pruning is to explore a “thin” or “shallower” version of it with similar accuracy while avoiding the exhaustive architecture search with heavy training processes. Thus the first value of pruning is to explore efficient architecture while avoiding heavy training. Therefore, it should be fast and efficient, ideally with no retraining or little fine-tuning. When the pruning method is too complex to implement or requires much more time than training from scratch, it could be an overkill and adds little value, especially when the performance is not better enough. Therefore, it is more informative if the authors would report the time/complexities for pruning/fine-tuning .
4. The second value of pruning lies at understand the redundancy of the model and providing insights for more efficient architecture designs.
5. Comparing to random initialization, pruning simply provide an initialization point inherited from the larger network. The essential question the author asked is whether a subset of pre-trained weights can outperform random initialization. This seems to be a common belief in transfer learning, knowledge distillation and the studies on initialization. The authors conclude that the accuracy of an architecture is determined by the architecture itself, but not the initialization. If this is true, training from scratch should have similar (but not better) result as fine-tuning a pruned model.  As the inherited weights can also be viewed as a “random” initialization. Both methods should reach equivalent good solution if they are trained with enough number of epochs. Can this be verified with experiments?
6. The experiments might not be enough to reject the common belief. The experiments only spoke that the pruned architectures can still be easily trained and encounter no difficulties during the optimization. One conjecture is that the pruned models in the previous work still have enough capacity for keeping good accuracy. What if the models are significantly pruned (say more than 70% of channels got pruned), is training from scratch still working well? It would add much value if the author can identify when training from scratch fails to match the performance obtained by pruning and fine-tuning.
7. In Section 4.1, “scratch-trained models achieve at least the same level of accuracy as fine-tuned models”. First, the ResNet-34-pruned A/B for this comparison does not have significant FLOPs reduction (10% and 24% FLOPs reduction). Fine-tuning still has advantage as it only takes ¼ of training time compare to scratch-E. Second, it is interesting that fine-tuning has generally smaller variance than stratch-E (except VGG-19). Would this imply that fine-tuning a pruned model produce more stable result? It would be more complete if there is variance analysis for the imagenet result.
8. What is the training/fine-tuning hyperparameters used in section 4.1?  Note that in the experiment of Li et al, 2017, scratch-E takes 164 epochs to train from scratch, while fine-tuning takes only 40 epochs. Like suggested above, if we fine-tune it with more epochs, would it achieve equivalent performance? Also, what is the hyperparameter used in scratch-E? Note that the original paper use batch size 128. If the authors adopts a smaller batch-size for scratch-E, then it has in more iterations and could certainly result in better performance according to recent belief that small batch-size generates better.
9. The conclusion of section 5 is not quite clear or novel. Using uniform pruning ratio for pruning is expected to perform worse than automatic pruning methods as it does not consider the importance difference of each layer and. This comes back to my point 3 & 4 about the value of pruning, that is the value of pruning lies at the analysis of the redundancy of the network. There are a number of works worked on analyzing the importance of different layers of filters. So I think the “hypothesis” of “the value of automatic pruning methods actually lies in the resulting architecture rather than the inherited weight” is kind of straightforward. Also, why not use FLOPs as x-axis in Figure 3?
Minor: It might be more accurate to use “L1-norm based Filter Pruning (Li et al., 2017)” as literally “channels” usually refers to feature maps, which are by-products of the model but not the model itself.
I  will revise my score if authors can address above concerns.
--------- review after rebuttal----------
#1#2 It would be great if the authors can make it clear that training is not the always the first step and the value of pruning in introduction rather than mentioning in conclusion. Saving training time is still an important factor when training from scratch is expensive.
#5 “fine-tuning with enough epochs”.
I understand that the authors are mainly questioning about whether training from scratch is necessarily bad than pruning and fine-tuning. The author do find that “training from scratch is better when the number of epochs is large enough”. But we see that fine-tuning ResNet-56 A/B with 20 epochs does outperform (or is equivalent to) scratch training for the first 160 epochs, which validates “fine-tuning is faster to converge”.  However, training 320 epochs (16x more comparing to 20 epochs fine-tuning and 2x comparing with normal training from scratch) is not quite coherent with the setting of “scratch B”, as ResNet-56 B just reduce 27% FLOPs.
The other part of the question is still unclear, i.e., the author claimed that the accuracy of an architecture is determined by the architecture itself, but not the initialization, then both fine-tuning and scratch training should reach equivalent solution if they are well trained enough, regardless of the initialization or pruning method. The learning rate for scratch training is already well known (learning rate drop brings boost the accuracy). However, learning rate schedule for fine-tuning (especially for significantly pruned model as for reply#6) is not well explored. I wonder whether that a carefully tuned learning rate/hyperparameters for fine-tuning may get the same or better performance as scratch training.
Questions:
- Are both methods using the same learning rate schedule between epoch 160 and epoch 320?
- The ResNets-56 A/B results in the reply#8 does not match the reported performance in reply#5. e.g., it shows 92.67(0.09) for ResNet-56-B with 40-epochs fine-tuning in reply5,  but it turns out to be 92.68(±0.19) in reply#8.
- It would be great if the authors can add convergence curves for fine-tuning and scratch training for easier comparison.
#6 The failure case for sparse pruning on ImageNet is interesting and it would be great to have the imageNet result reported and discussed.
The authors find that “when the pruned ratio is large enough, training from scratch is better by a even larger margin than fine-tuning”.  This could be due to following reasons:
      1. When the pruning ratio is large, the pruned model with preserved weights is significantly different from the original model, and fine-tuning with small learning rate and limited number of epochs is not enough to recover the accuracy. As mentioned earlier, tuning the hyperparameters for fine-tuning based on pruning ratio might improve the performance of fine-tuning.
      2. Though the pruning ratio is large, the model used in this experiment may still have large capacity to reach good performance. How about pruning ResNet-56 with significant pruning ratios?
Finally, based on above observations, it seems to me that the preserved weights is more essential for fast fine-tuning but less useful for significant pruning ratios.
-------- update ----------------
The authors addressed most of my concerns. Some questions are still remaining in my comment “Review after rebuttal”,  specifically, fine-tuning a pruned network may still get good performance if the hyperparameters are carefully tuned based on the pruning ratios, or in other words, the preserved weights is more essential for fast fine-tuning but less useful for significant pruning ratios. The authors may need to carefully made the conclusion from the observations. I would hope the authors can address these concerns in the future version.
However, I think the paper is overall well-written and existing content is inspiring enough for readers to further explore the trainability of the pruned network. Therefore I raised my score to 7.


Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any derived quantities, but can be shown to be equivalent to maximizing likelihood under some conditions. Our result holds in the non-asymptotic parametric setting, where both the capacity of the model and the number of data examples are finite. We also demonstrate encouraging experimental results.

Two high-level points about my review before going into the details:
1. This paper was a thoroughly enjoyable and insightful read. Kudos to the authors for attempting such a comprehensive overview of likelihood-based vs. likelihood-free learning.
2. I’ll be more than happy to revise my current rating if my concerns are addressed by the authors.
With regards to the technical assessment of this work, the idea of using a nearest neighbors objective for learning a generative model is both intriguing and appealing. What makes this work even more interesting are its connections with maximum likelihood estimation. Novelty aside, I believe there are major theoretical, algorithmic, and empirical concerns in the current work which I discuss below:
Theorem 1
- The third condition is true for location-scale family of distributions e.g., Gaussian. But the distribution learned by a generative model p_theta is far from Gaussian or other location-scale distributions.
- More importantly, I don’t think the upper bound is tight in practice because the likelihoods can vary significantly across the dataset. Take MNIST for example. Compare the log-likelihoods of an autoregressive model or ELBOs of a VAE across the different classes of digits. Straight digits (like 1s) have much higher log-likelihoods on average than curved digits.
Algorithm
- While significant advancements have indeed been made for nearest neighbor evaluation as the authors highlight, it’s hard to believe without any empirical evidence that nearest neighbor evaluation is indeed efficient in comparison to other methods of likelihood evaluation.
- Similarly, I was a bit disappointed by the choice of Euclidean distance in a pixel space as the choice of distance metric. The argument that you do not want to use “auxiliary sources of labelled data or leverage domain-specific prior knowledge” is indeed necessary for fair comparisons, but also points to a limitation of the current approach.
Empirical evaluation
- Seems too outdated both in terms of baselines and metrics. The authors are clearly aware of the current research in generative modeling but the current work provides almost no strong evidence to consider this work as an alternative to other approaches.
- While it is arguably well-established that Parzen window estimates are misleading (Theis et al.), that’s the only quantitative estimate in this work (Table 1). Hard to think of any recent published work (last 1-2 years) in generative modeling that even reports these estimates.
- The baselines in Table 1 are all from 2013-15. Clearly, much has happened in the last 3 years that merit the inclusion of more recent baselines.
-  Even for sample quality, there has been a lot of research in designing and improving metrics. E.g., Inception scores, Frechet Inception Distance, Kernel Inception Distance. I am not looking for state-of-the-art numbers, showing heavily zoomed out samples without any of these metrics is slightly disingenuous.
- As mentioned before, reporting the computation time/per iteration and number of iterations for convergence for the proposed algorithm in comparison with other approaches  is important.
- Similarly, the argument about the method avoiding even the other GAN problems (e.g., vanishing gradients, stability in training) can and should be supported by empirical evidence.
Analysis and discussion
- One family of generative models that is crucially missing from this work is normalizing flow models.
- This is somewhat debatable, but I do not agree that the tradeoff between likelihoods and sample quality is due to model capacity. As far as I can tell, the cited work of Grover et al., 2017 provides evidence contrary to what the authors claim. The prior work trained the same normalizing flow model via maximum likelihood and adversarial training, and observed vastly different results on likelihood and sample quality metrics. So model capacity isn't necessarily the key differentiating factor (which is same for both training algorithms in their experiments), it's more about the choice of the objective function and the optimization procedure.
Minor points for improving presentation:
- Section 3 can be made more concise and to the point. I’d be especially interested if the precision and recall discussion in this section and elsewhere can be formalized.
- Use numbered lists instead of bullets for assumptions in Theorem 1, so that the discussion of the assumptions right after the theorem statement are easy to follow.
- The citation of Grover et al. seems outdated? The current title is Flow-GAN: Combining maximum likelihood and adversarial learning in generative models.
- In general, avoid making somewhat hard assertions that are speculative. Some of them I’ve highlighted earlier in my review (e.g., some of the theorem assumptions being typically true, comparison of likelihood and sample quality based on model capacity etc.).


Reinforcement learning (RL) often requires large numbers of trials to solve a single specific task. This is in sharp contrast to human and animal learning: humans and animals can use past experience to acquire an understanding about the world, which they can then use to perform new tasks with minimal additional learning. In this work, we study how an unsupervised exploration phase can be used to build up such prior knowledge, which can then be utilized in a second phase to perform new tasks, either directly without any additional exploration, or through minimal fine-tuning. A critical question with this approach is: what kind of knowledge should be transferred from the unsupervised phase to the goal-directed phase? We argue that model-based RL offers an appealing solution. By transferring models, which are task-agnostic, we can perform new tasks without any additional learning at all. However, this relies on having a suitable exploration method during unsupervised training, and a model-based RL method that can effectively utilize modern high-capacity parametric function classes, such as deep neural networks. We show that both challenges can be addressed by representing model-uncertainty, which can both guide exploration in the unsupervised phase and ensure that the errors in the model are not exploited by the planner in the goal-directed phase. We illustrate, on simple simulated benchmark tasks, that our method can perform various goal-directed skills on the first attempt, and can improve further with finetuning, exceeding the performance of alternative exploration methods.

The authors address the problem of how to use unsupervised exploration in a first phase of reinforcement learning to gather knowledge that can be transferred to new tasks to improve performance in a second task when specific reward functions are available. The authors proposed a model-based approach which uses deep neural networks as a model for the environment. The model is PETS (probabilistic ensembles with trajectory sampling), an ensemble of neural networks whose outputs parametrize predictive distributions for the next state as a function of the current state and the action applied. To collect data during the unsupervised exploration phase, they use a metric of model uncertainty computed as follows: the average over all the particles assigned to each bootstrap is computed and the variance over these computed means is the
metric of uncertainty. The authors validate their method on the HalfCheetah OpenAI gym environment where they consider 4 different tasks related to running forward, backward, tumbling forward and tumbling backward. The results obtained show that they outperform random and count based exploration approaches.
Quality:
I am concerned about the quality of the experimental evaluation of the method. The authors only consider a single environment for their experiments and artificially construct 4 relatively similar tasks. I believe this is insufficient to quantify the usefulness of the proposed method.
Clarity:
The paper is clearly written and easy to read.
Novelty:
The proposed approach seems incremental and lacks novelty. The described method for model-based exploration consists in looking at the mean of the prediction of each neural network in the ensemble and then computing the empirical average. This approach has been used before for active learning with neural networks ensembles:
Krogh, Anders, and Jesper Vedelsby. "Neural network ensembles, cross validation, and active learning." Advances in neural information processing systems. 1995.
The used model, PETS, is also not novel and the proposed methodology for having first an unsupervised learning phase and then a new specific learning task is also not very innovative.
Significance:
Given the lack of a rigorous evaluation framework and the lack of novelty of the proposed methods, I believe the significance of the contribution is very low.


FEATURE INTERTWINER FOR OBJECT DETECTION
A well-trained model should classify objects with a unanimous score for every category. This requires the high-level semantic features should be alike among samples, despite a wide span in resolution, texture, deformation, etc. Previous works focus on re-designing the loss function or proposing new regularization constraints on the loss. In this paper, we address this problem via a new perspective. For each category, it is assumed that there are two sets in the feature space: one with more reliable information and the other with a less reliable source. We argue that the reliable set could guide the feature learning of the less reliable set during training -in the spirit of student mimicking teacher's behavior and thus pushing towards a more compact class centroid in the high-dimensional space. Such a scheme also benefits the reliable set since samples become closer within the same category -implying that it is easier for the classifier to identify. We refer to this mutual learning process as feature intertwiner and embed the spirit into object detection. It is well-known that objects of low resolution are more difficult to detect due to the loss of detailed information during network forward pass. We thus regard objects of high resolution as the reliable set and objects of low resolution as the less reliable set. Specifically, an intertwiner is achieved by minimizing the distribution divergence between two sets. We design a historical buffer to represent all previous samples in the reliable set and utilize them to guide the feature learning of the less reliable set. The design of obtaining an effective feature representation for the reliable set is further investigated, where we introduce the optimal transport (OT) algorithm into the framework. Samples in the less reliable set are better aligned with the reliable set with aid of OT metric. Incorporated with such a plug-and-play intertwiner, we achieve an evident improvement over previous state-of-the-arts on the COCO object detection benchmark.

OVERVIEW:
The authors tackle the problem of detecting small/low resolution objects in an image. Their key idea is that detecting bigger objects is an easier task and can be used to guide the detection of smaller objects. This is done using the "Feature Intertwiner"  which consists of two branches, one for the larger objects (more reliable set that is also easier to detect) and one for the smaller objects (less reliable set). The second branch contains a make-up layer learned during training (which acts as the guidance from the more reliable set) that helps compensate details needed for detection. The authors define a class buffer that contains representative elements of object features from the reliable set for every category & scale and an intertwiner loss that computes the L2 loss between the features from the less reliable set & the class buffer. They also use an Optimal Transport procedure with a Sinkhorn divergence loss between object features from both sets. The overall loss of the system is now a sum of the detection loss, the intertwiner loss and the optimal transport loss. They evaluate their model on the COCO Object detection challenge showing state-of-the-art performance. They also provide thorough ablation analysis of various design choices. The qualitative result in Fig.1 showing well clustered features for both high & low resolution objects via t-SNE is a nice touch.
COMMENTS:
Clarity - The paper is well written and easy to follow.
Originality & Significance - The paper tackles an important problem and provides a novel solution.
Quality - The paper is complete in that it tackles an important problem, provides a novel solution and demonstrates via thorough experiments the improvement achieved using their approach.
QUESTIONS:
1. The Class Buffer seems very restricted in having a single element per object category per scale to represent all features. The advantage of forcing such a representation is tight clustering in the feature space. But, wouldn't a dictionary approach with multiple elements give more flexibility to the model and learn a richer feature representation at the cost of not-so-good clustering ?
2. Any comment on why you drop performance for couch ? (and baseball bat + bedroll)
3. In Table 4 of Appendix where you compare with more object detection results, I find it interesting that Mask RCNN, updated results has a might higher AP_S (43.5) compared to you (27.2) and everyone else. I was expecting you to be the best under that metric due to the explicit design for small objects. They (MaskRCNN, updated results) are also significantly better than the rest under AP_M but worse under AP_L. Can you explain this behavior ? Is the ResNeXt backbone that much better for small objects ?


Recently, generative adversarial network is the hotspot in research areas and industrial application areas. Its application on data generation in computer vision is most common usage. This paper extends its application to data hiding and security area. In this paper, we propose the novel framework to integrate steganography and steganalysis processes. The proposed framework applies generative adversarial networks as the core structure. The discriminative model simulate the steganalysis process, which can help us understand the sensitivity of cover images to semantic changes. The steganography generative model is to generate stego image which is aligned with the original cover image, and attempts to confuse steganalysis discriminative model. The introduction of cycle discriminative model and inconsistent loss can help to enhance the quality and security of generated stego image in the iterative training process. Training dataset is mixed with intact images as well as intentional attacked images. The mix training process can further improve the robustness and security of new framework. Through the qualitative, quantitative experiments and analysis, this novel framework shows compelling performance and advantages over the current state-of-the-art methods in steganography and steganalysis benchmarks.

The paper is on applying GAN to steganography, and it is especially suited for people who does not know steganography. In fact, 1/4 of the paper is spent on introducing (in a very clear and catchy way) the basics. State of the art section nonetheless shows that the authors are grounded with the very last related work. The novel GAN framework is called ISS-GAN. Simply speaking, ISS-GAN is built by the combo: <steganography generative, steganalysis discriminative>. In the details, the implementation is not straightforward, and embeds interesting ideas. Two generative and two discriminative models are intertwined: one generative for the stego image generation process, the other generative for the secret image extraction process. The two discriminative models are for ensuring that the distribution of cover images is indistinguishable from the stego distribution and to ensure that the distribution of extracted secret images is indistinguishable from the distribution of the original secret images. In rough terms, the adversarial double loop serves to ensure that the embedding and the extraction function preserve both the aspect of the original and the secret images. PSNR and SSIM metrics are employed to give a quantitative check of the approach.
Obvious questions are about the scalability of the approach, since it applies to 256x256 miniatures, which act well on the figures of an ICLR paper but poor in reality. At the same time, is it reasonable to show images such that Figure 3? It is obvious stega processes are now very effective, and stressing that by showing seemingly same images it is not very illustrative. Figure 4 typos are reated to row column ordering, but even in this case is hard to spot something without spending time in magnifying the pdf playing at look at the differences… Figure 5 and 6 are even worse, with tens of practically equal images. Less images with differences pictures should have been better. In addition, with these qualitative results, it is hard to give a weight to the quantitative comparisons, in terms of both the PSNR and SSIM metrics.
The most obvious and perhaps elementary question I have is how risky is that the ISS GAN model can be replicated by an attacker which sniffs the secret images by simply having a training set which approximatively is the same of the primary sender?
The other questions are about how much realistic is that the images are affected by the noises of Table 5, nowadays?


Recent years have witnessed two seemingly opposite developments of deep convolutional neural networks (CNNs). On the one hand, increasing the density of CNNs (e.g. by adding cross-layer connections) achieves higher accuracy. On the other hand, creating sparsity structures (e.g. through regularization and pruning methods) enjoys a more slim network structure. In this paper, we bridge these two by proposing a new network structure with locally dense yet externally sparse connections. This new structure uses dense modules as basic building blocks and then sparsely connects these modules via a novel algorithm during the training process. Experimental results demonstrate that the locally dense yet externally sparse structure could obtain competitive performance on benchmark tasks (CI-FAR10, CIFAR100,and ImageNet) while keeping the network structure slim.
Under the inspiration of bridging these two trends and search more efficient network structures, our paper explores methods which directly introduce sparsity into network structure thus avoid pruningafter-training strategy. In neural science, papers (e.g. (Betzel et al., 2017;Sztarker & Tomsic, 2011;Gazzaniga, 1988)) concentrating on the brain structure reveal that neuron connections in brain perform a locally dense but externally sparse property as paper (Betzel et al., 2017) shown, that the closer two regions are, the denser the connections between them will be. Visual cortex papers (Belliveau et al., 1991) show that while sensory information arrives at the cortex, it is fed up through hierarchy regions from primary area V1 up to higher areas such as V2, V4 and IT. Inside of each cor-tex layer, tightly packed pyramidal cells consist of basic locally dense structures in the brain. While our brain has been trained over time, internal densely connected modules will form a few long distance and cross-level connections to transfer information to higher hierarchy. Modular structures have shown vital importance in our brain behaviors such as specializing in information processing (Espinosa-Soto & Wagner, 2010), performing focal functions (Baldassano & Bassett, 2016), and supporting complex neural dynamics.
In this case, instead of creating local density by pruning redundancy on the trained model, we perform local density by prefixing untrained dense modules as tightly packed neuron cell in the human brain and let it evolving both the weights of itself and the sparse connection between them via training. Since DenseNet has reached theoretical densest connection status, we use a similar dense block structure with growth rate k, but only with very narrow channels in each block. The growth rate k (Huang et al. (2017)) is a hyper parameter in Densely Connected structures, which denotes growth rate of the input feature map scale when network goes deeper. Previous methods constructing neural modules with structural sparsity (e.g. (Xie et al., 2018;Szegedy et al., 2017)) are mostly empirically constructing the sparse connection between modules. To give more convincing guidance of forming sparse connections, we design a genetic training strategy to search an optimized connection matrix. This algorithm treats the connection matrix as the gene, and only reserves mutated individual with the best performance among others. Actually, this strategy consistently changes the input feature groups during training process, and by always counting new feature distribution in, this strategy could take similar effect as drop-out methods, thus make the model robust. Moreover, besides merely creating parallel connections between modules, our algorithm could create long-distance connections between input module and output module by a transit layer.
The experiment results demonstrate that evolving locally dense but externally sparse connections could maintain competitive performance on benchmark image datasets while using compared slim network structures. By comparison experiments, we reveal contribution proportion on the final performance of each specific connection, and by that give the principle of design sparse connections between dense modules. The main contribution of this paper is as follows:
• We enhance the hierarchical structure by utilizing the property of locally dense but externally sparse connections.
• Instead of empirically constructing module connections, we design an evolving training algorithm to search optimized connection matrix.
• We let each module choose output flow globally rather than simply creating parallel streams between modules so that the feature could flow to final layer through various depth.
• We give a detailed analysis of how different sparse connections and different module properties will contribute to the final performance. Moreover, We reveal contribution proportion on the final performance of each connection and each module by several contrast experiments, and by that give principle of design sparse connections between dense modules.

The authors present the interesting and important direction in searching better network architectures using the genetic algorithm. Performance on the benchmark datasets seems solid. Moreover, the learned insights described in Section 4.4 would be very helpful for many researchers.
However, the overall paper needs to be polished more. There are two many typos and errors that imply that the manuscript is not carefully polished. Explanations about some terms like growth rate, population, etc. are necessary for broader audience.
More importantly, while some of step jumps in Figure 6~9 are suspicious, it turns out that all the step jumps happen at the same number of steps, which are identical to the change of learning rates described in Section 4.2. Thee clear explanation about that phenomena is required.
* Details
- Please represent the blocks (e.g. 1*1conv) better. Current representation is quite confusing to read. Maybe proper spacing and different style of fonts may help.
- In Page 5, "C_{m}ax" is a typo. It should be "C_{max}".
- Regarding the C_max, does sum(C_max) represent (D * W)^2 where D is the total depth and W is the total indicies in each layer? If so, specifying it will help. Otherwise, please explain its meaning clearly.
- In Figure 4(a), it would be better if we reuse M_{d,w} notation instead of Module {d_w}.
- Please briefly explain or provide references to the terms like "growth rate", "population", and "individuals".
- Different mutations may favor different hyper-parameters. How the authors control the hyperparameters other than the number of epochs will be useful to know.
- Even though the sparse connection is enforced for some reasons, overfitting, variance, or any other benefits that slim structure can bring in has not been evaluated. They need to be presented to verify the hypothesis that the authors claim.


We construct a general unified framework for learning representation of structured data, i.e. data which cannot be represented as the fixed-length vectors (e.g. sets, graphs, texts or images of varying sizes). The key factor is played by an intermediate network called SAN (Set Aggregating Network), which maps a structured object to a fixed length vector in a high dimensional latent space. Our main theoretical result shows that for sufficiently large dimension of the latent space, SAN is capable of learning a unique representation for every input example. Experiments demonstrate that replacing pooling operation by SAN in convolutional networks leads to better results in classifying images with different sizes. Moreover, its direct application to text and graph data allows to obtain results close to SOTA, by simpler networks with smaller number of parameters than competitive models.

Summary:
The paper argues that plain (fully connected) neural networks cannot represent structured data, e.g. sequences, graphs, etc. Specialized architectures have instead been invented for each such case, e.g. recurrent neural networks, graph networks etc. The paper then proposes to treat these structured data types as sets and propose a neural network method for converting a set into a fixed size vector, which can then be classified using plain neural networks. The method works by projecting each member of the set into M dimensions where they are passed through a ReLU and summed. The paper proves that given high enough dimensionality M, no information will be lost during this process. The paper performs experiments on graph data, text data and image data. The paper concludes "We proposed a general framework for processing structured data by neural network."
Quality:
The paper seems rushed. I think the paper has some language problems which unfortunately detract from the overall impression. "through its all one-dimensional projections". Should that be "through all its one-dimensional projections"? "Analogically". "can lead to the >lose< of meaningful information". "where (the?) Radon transform".
Clarity:
The aim and purpose of the paper is fairly clear. I think the method explanation is overly complicated. If I understand correctly, the proposed method is simply to map each set element to an M dimensional space using a linear projection followed by a ReLU and then summing the set elements. The section on how to implement it in practice further complicates the paper unnecessarily in my opinion. It's not clear why or whether the ReLU is important.
Originality:
Transforming sets into fixed size vectors is not new. See e.g. [1] and [2], which the paper does not reference or compare to.
To the papers defence I think the main idea is to map any structured data into a set, and then use a (relatively) simple method on it.
Significance:
The paper acknowledges that methods do indeed exist for the various structured data types, but claim that they are complex, and that the proposed method is a simple general alternative. As such the significance of the paper hinges on whether the proposed method is indeed simpler, and how it compares when it comes to performance. The proposed method is simple and general, but it does require that the information lost when converting the structured data to the set is encoded as features in the set elements, e.g. the sequence index is added. For images, the normalized position must be added. For graphs, the edges should be added, which interestingly is not done in the single graph experiment (I'd be curious how the edges could be added). This detracts somewhat from the claim to generality, since each structured data type must still be handled differently.
It's clear to me that the structural priors built into the specialized networks, e.g. recurrent, graph, etc. should help for these data structures. For this reason I think the proposed method will have a hard time comparing head to head. That is OK, it becomes a tradeoff between generality versus performance. Unfortunately I'm not convinced of the performance by the experiments. Specifically the proposed method is not compared head to head against the methods it proposes to replace.
 * On the graph data it is compared against various classifiers that use a fixed size, handcrafted representation. This is disingenuous in my opinion. For graph data, the natural method it proposed to replace is a graph neural network.
 * On the sequence data it is compared against a 1D convolutional neural network. The canonical sequence model is a recurrent neural network. Also for the larger IMDB dataset the performance drop against a CNN is considerable.
 * The image experiments are probably the strongest case. Here the authors make a compelling case that the SAN pooling operator is better than max-pooling. Unfortunately the authors use a dataset which have already been size normalized during pre-processing, and then un-normalize it. It'd be more convincing if the authors showed superior performance on a dataset of images of varying sizes using SAN, compared to normalizing them during pre-processing (with comparable runtime and parameter counts).
Pros:
 - Ambitious aim
 - Simple method
Cons:
 - Proposing to replace xyz methods, then not fairly comparing to them in their respective domains.
 - Not referencing relevant prior work on neural networks for sets.
 - Paper seems rushed/unfinished
 - Implicitly assumes classification/regression for entire structured data. In many cases, e.g. Named Entity Recognition, and many graph problems, the problem is to classify/regress on the individual elements, as part of the whole.
I could see the paper accepted, if sold less as the end-all solution for structured data and more as simply an improved pooling operator, with SOTA experiments to back up the claims. This would also allow the authors to use SOTA methods for pre-processing the data, e.g. RNNs, graph neural networks, etc., as they don't need to compete against every method.
(btw. instead of zero padding the batches, the authors can probably maintain a list of indices of set elements and use https://www.tensorflow.org/api_docs/python/tf/math/segment_sum)
[1] - Vinyals, Oriol, Samy Bengio, and Manjunath Kudlur. "Order matters: Sequence to sequence for sets." arXiv preprint arXiv:1511.06391 (2015).
[2] - Zaheer, Manzil, et al. "Deep sets." Advances in Neural Information Processing Systems. 2017.


Serial crystallography is the field of science that studies the structure and properties of crystals via diffraction patterns. In this paper, we introduce a new serial crystallography dataset comprised of real and synthetic images; the synthetic images are generated through the use of a simulator that is both scalable and accurate. The resulting dataset is called DiffraNet, and it is composed of 25,457 512x512 grayscale labeled images. We explore several computer vision approaches for classification on DiffraNet such as standard feature extraction algorithms associated with Random Forests and Support Vector Machines but also an end-to-end CNN topology dubbed DeepFreak tailored to work on this new dataset. All implementations are publicly available and have been fine-tuned using off-the-shelf AutoML optimization tools for a fair comparison. Our best model achieves 98.5% accuracy on synthetic images and 94.51% accuracy on real images. We believe that the DiffraNet dataset and its classification methods will have in the long term a positive impact in accelerating discoveries in many disciplines, including chemistry, geology, biology, materials science, metallurgy, and physics.

Contribution:
	- Using a known parameters crystallography simulator (X-ray beam, structure being analyzed, environment (crystalline or not)) built a dataset (called DiffraNet) of 25,000 512x512 grayscale labeled images of resulting diffraction images of various materials/structures (crystalline or not) .
	- carried various classification approaches of the dataset (labelled) images in two steps:
		- Feature extraction (Scale Invariant Feature Transform with the Bag-of-Visual-Words approach as local feature extractor, and the Gray-level Co-occurrence Matrix and Local Binary Patterns as global feature extractor) then
		- Classification of the diffraction images is carried with three approaches. Two using images described by extracted features (from the previous feature extraction step) coupled with either random Forests or Support Vector Machines and a third consisting in a Convolution Neural Network (CNN) topology named DeepFreak.
		- The images are classified according to the diffraction patterns they encompass into one of 5 classes: blank, no-crystal, weak, good and strong. The last three describing presence of a crystalline structure.
		- A fine tuning step of the various algorithms was carried using AutoML optimization tools.
All algorithms were off the shelf publicly available implementations and have previously been used for such domain applications (crystallography patterns).
The approach and choices of classification algorithms is well articulated and results interesting.
A few questions though:
•	In what way the diffraction images are ‘synthetic’? Aren’t they actual diffraction images but in a controlled known and controlled setting: set of parameters (beam, structure to analyze)?
o	More like a library of diffraction pattern images for various materials/structures.
•	How many structures were analyzed (Were there 25000 for the 25000 pattern images), one image each?
o	This is to understand  the representability of the samples (structures) analyzed regarding the possible structures (Hundreds of thousands as per paper’s 2.1 ) .
•	What variations for each of the setting variabilities (X Ray beam(flux, beam size, divergence, and bandpass), crystal properties (unit cell, number of cells, and a structure factor table), and the experimental parameters (sources of background noise, detector point-spread, and shadows)) were used?
o	This is to assess the size of the pattern space.
•	Were any real-life setting obtained pattern samples classified using DiffraNet dataset patterns’ fine-tuned classification algorithms?
o	This is to assess the generalization level of the DiffraNet dataset patterns’ fine-tuned classification algorithms to real-life obtained patterns (relates to the previously stated representability of the samples).
o	If not, your statement “ … we plan to add new images and new classes that are common place in serial crystallography” (in 6. Conclusions) would be an appreciated validation of general usability of your DiffraNet fine–tuned setting.
•	Were all the structures analyzed crystalline?
o	It’s stated in Figure 2 and Table 6 that 2 classes are either blank or no-crystal but is that a known fact (purposely chosen) or no pattern images for crystalline structures due to inadequate experimental settings to uncover the crystalline nature of the analyzed structure?
•	Were the pattern images pre-processed in any manner before being classified?
Nota: In table 6, use no-crystal class as in Figure 2 for consistency.


The meaning of a sentence is a function of the relations that hold between its words. We instantiate this relational view of semantics in a series of neural models based on variants of relation networks (RNs) which represent a set of objects (for us, words forming a sentence) in terms of representations of pairs of objects. We propose two extensions to the basic RN model for natural language. First, building on the intuition that not all word pairs are equally informative about the meaning of a sentence, we use constraints based on both supervised and unsupervised dependency syntax to control which relations influence the representation. Second, since higher-order relations are poorly captured by a sum of pairwise relations, we use a recurrent extension of RNs to propagate information so as to form representations of higher order relations. Experiments on sentence classification, sentence pair classification, and machine translation reveal that, while basic RNs are only modestly effective for sentence representation, recurrent RNs with latent syntax are a reliably powerful representational device.

The paper presents an extension of relation networks (RNs) for natural language processing. RNs are designed to represent a set as a function of the representations of their elements. This paper treats a sentence as a set of words. Whereas regular RNs assume that the representation of a set is a uniform aggregation of the representation of the pairs of elements in the set, this paper proposes to weight the relevance of the pairs according to their tree-structured dependency relations between the words. The authors evaluate on a suite of NLP tasks, including SNLI, Quora duplicate question ID, and machine translation. They show marginal improvements over naive baselines, and no improvement over SOTA.
I am concerned about both the motivation for and the novelty of this work. My reading of this work is that the authors try to reverse engineer a TreeRNN in terms of RNs, but I am not sure what the reason is for wanting to use the RN framework in order to derive an architecture that, IIUC, essentially already exists. I can't find any fundamentally meaningful differences between the proposed architecture and the existing work on TreeRNNs, and the results suggest that there is nothing groundbreaking being proposed here. It is possible I am missing some key insight, but I do believe the burden is on the authors to highlight where the novelty is. The intro *and* related work sections should both be rewritten to answer the question: what is the insufficiency with current sentence encoding models that is addressed by this architecture? Currently, the intro addresses the tangential question: what is the insufficiency with RNs for NLP that is addressed by this architecture? If the latter is the question the authors want to answer, they need to first answer: why should we want to cast sentence encoders as RNs as opposed to any of the (many) other available architectures? Without a firmer understanding of what this paper contributes and why, I can't recommend acceptance. More detailed comments for the authors below.
- You introduce a few naive baselines, but none of these is a TreeRNN. TreeRNNs are the obvious baseline, and you should be comparing on each and every evaluation task, even if there is no previously published result for using tree RNNs on that task. For the one result (SNLI, table 1) on which there is previous work using TreeRNNs, the table confirms my intuition that the proposed model is no improvement over the TreeRNN architecture. It seems very important to address this comparison across all of the evaluation tasks.
- I like the notion of marginalizing over latent tree structures, but the related work section needs to make clear what is being contributed here that is different from the cited past work on this problem
- On the MT eval, why are you missing values for zh-en on the NMT models that are actually competitive? I think many of these models are open-source or easy to reimplement? Its hard to draw conclusions when from such a gappy table.
- Only table 2 has significance values (over naive baseline that is) which implies that the other results are not significant? That is disconcerting.
- I am disappointed in the analysis section. As is, you provide an ad-hoc inspection of some inferred trees. I find this odd since there is no evidence that the tree-ness of the architecture (as opposed to, e.g., recurrence or attention) is what leads to quantitative improvements (at least according to the experimental results in the tables), so there is no reason we should actually expect the trees to be good or interesting. My interpretation of these cherry-picked examples is that the learning is fighting the architecture a bit, basically "learning a tree" that reduces to being an attention mechanism that up-weights one or two salient words.
- The analysis I *wanted* to see instead was why recursion helped for sentence classification, it did not for MT. You give an intuition for this result but no evidence. (That is assuming that, quantitatively, this trend actually holds. Which maybe is not the case if none of the results are significant.)
- In general, regarding evaluation, SNLI is overfit. You should use MNLI at least. I have trouble interpreting progress on SNLI as "actual" progress on language representation.
- The related work section as a whole is too short. If you need to cut space, move technical content to appendix, but don't compromise in related work. You listed many relevant citations, but you have no context to situate your contribution relative to this past work. What is the same/different about your method? You should provide an answer to that for each and every paper you cite.


Generating coherent and cohesive long-form texts is a challenging problem in natural language generation. Previous works relied on a large amount of humangenerated texts to train neural language models, however, few attempted to explicitly model the desired linguistic properties of natural language text, such as coherence and cohesion using neural networks. In this work, we train two expert discriminators for coherence and cohesion to provide hierarchical feedback for text generation. We also propose a simple variant of policy gradient, called negative-critical sequence training in which the reward baseline is constructed from randomly generated negative samples. We demonstrate the effectiveness of our approach through empirical studies, showing improvements over the strong baseline -attention-based bidirectional MLE-trained neural language model -in a number of automated metrics. The proposed model can serve as baseline architectures to promote further research in modeling additional linguistic properties for downstream NLP tasks.

The paper proposes a method for improving the quality of text generation by optimizing for coherence and cohesion. The authors develop two discriminators--a "coherence discriminator" which takes as input all of the sentence embeddings (i.e. averaged word embeddings) of the document and assigns a score, and a "cohesion discriminator" which takes as input the word embeddings of two consecutive sentences and assigns a score. In the former, the score is the cosine similarity between the encodings of the first and second half of the document. In the latter, the score is the cosine similarity between the encodings of the two sentences. Both discriminators use CNNs to encode the inputs. The discriminators are trained to rank true text over randomly drawn negative samples, which consist of randomly permuted sentence orderings and/or random combinations of first/second half of documents. This discriminators are then used to train a text generation model. The output of the text generation model is scored by various automatic metrics, including NLL, PPL, BLEU, and number of unique ngrams in the outputs. The improvements over a generically-trained generation model are very small.
Overall, I did not find this paper to be convincing. The initial motivation is good--we need to find a way to capture richer linguistic properties of text and to encourage NLG to produce such properties. However, the discriminators presented do not actually capture the nuances that they purport to capture. As I understand it, these models are just being trained to incentivize high cosine similarity between the words in the first/second half of a document (or sentence/following sentence). That is not reflective of the definitions of coherence and cohesion, which should reflect deeper discourse and even syntactic structure. Rather, these are just models which capture topical similarity, and naively at that. Moreover, training this model to discriminate real text from randomly perturbed text seems problematic since 1) randomly shuffled text should be trivially easy to distinguish from real text in terms of topical similarity and 2) these negative samples are not (I don't think) at all reflective of the types of texts that the discriminators actually need to discriminate, i.e. automatically generated texts. Thus, even ignoring the fact that I disagree with the authors on exactly what the discriminators are/should be doing, it is still not clear to me that the discriminators are well trained to do the thing the authors want them to do. I have various other concerns about the claims, the approach, and the evaluation. A list of more specific questions/comments for the authors is below.
- There are a *lot* of unsubstantiated claims and speculation about the linguistic properties that these discriminators capture, and no motivation of analysis as to how they are capturing it. Claims like the following definitely need to be removed: "learn to inspect the higher-level role of T, such as but not limited to, whether it supports the intent of S, transitions smoothly against S, or avoids redundancy", "such as grammar of each of the sentences and the logical flow between arbitrary two consecutive sentences"
- You only use automated metrics, despite acknowledging that there is no good way to evaluate generation. Why not use human eval? This is not difficult to carry out, and when you are arguing about such subtle properties of language, human eval is essential. There is no reason that BLEU, for example, would be sensitive to coherence or cohesion, so why would this be a good way to evaluate a model aimed to capture exactly those things?
- Also related to human eval, there should be an intrinsic evaluation of the discriminators. Do they correlate with human judgments of coherence and cohesion? You cannot take it for granted that they capture these things (I very much believe they do not), so present some evidence that the models do what you claim they do.
- The reported improvements are minuscule, to the extent that I would read them as "no difference". The only metric where there is a real difference is on number of unique ngrams generated cross inputs, which is presumably because its just learning (being encouraged to) spit out words that were in the input. I'd like to see the baseline of just copying the input as the output.
- You mention several times that these models will pick up on redundancy. It is not clear to me how they could do that. Aren't they simply using a cosine similarity between feature vectors? Perhaps I am missing something, but I don't see how this could learn to disincentivize redundancy but simultaneously encourage topical similarity. Could you explain this claim?


Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution. Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint. In this study, we prove the local stability of optimizing the simple gradient penalty µ-WGAN(SGP µ-WGAN) under suitable assumptions regarding the equilibrium and penalty measure µ. The measure valued differentiation concept is employed to deal with the derivative of the penalty terms, which is helpful for handling abstract singular measures with lower dimensional support. Based on this analysis, we claim that penalizing the data manifold or sample manifold is the key to regularizing the original WGAN with a gradient penalty. Experimental results obtained with unintuitive penalty measures that satisfy our assumptions are also provided to support our theoretical results.

In the paper, WGAN with a squared zero centered gradient penalty term w.r.t. to a general measure is studied. Under strong assumptions, local stability of a time-continuous gradient ascent/descent dynamical system near an equilibrium point are proven for the new GP term. Experiments show comparable results to the original WGAN-GP formulation w.r.t. FID and inception score.
Overall, I vote for rejecting the paper due to the following reasons:
- The proven convergence theorem is for a time-continuous "full-batch" dynamical system, which is very far from what happens in practice (stochastic + time discrete optimization with momentum etc). I don't believe that one can make any conclusions about what is actually happening for GANs from such an idealized setting. Overall, I don't understand why I should care about local stability of that dynamical system.
- Given the previous point I feel the authors draw too strong conclusions from their results. I don't think Theorem 1 gives too many insights about the success of gradient penalty terms.
- There are only marginal improvements in practice over WGAN-GP when using other penalty measures.
Further remarks:
- In the introduction it is claimed that mode collapse is due to JS divergence and "low-dimensionality of the data manifold". This is just a conjecture and the statement should be made more weak.
- The preliminaries on measure theory are unnecessarily complicated (e.g. partly developed in general metric spaces). I suggest that the authors try to simplify the presentation for the considered case of R^n and avoid unnecessarily complicated ("mathy") definitions as they distract from the actual results.
==after rebuttal==
After reading the authors rebuttal I increased the my rating to 6 as they addressed some of my doubts. I still think that the studied setting is too idealized, but it is a first step towards an analysis.


A CLOSER LOOK AT DEEP LEARNING HEURISTICS: LEARNING RATE RESTARTS, WARMUP AND DISTILLA- TION
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz., mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.

Summary:
This paper uses the recently proposed techniques of mode connectivity and CCA to analyze two different popular heuristics in deep learning:
(1) SGDR (stochastic gradient descent with restarts/cosine annealing of learning rate)
(2) Learning rate warmup
(3) Model distillation
For (1) they visualize 1d and 2d slices of the loss surface either using mode connectivity, or parameter points immediately before restarts to try and understand if the parameters sit in different local minima. For (2), they study the effect of learning rate warmup using CCA, coming to the conclusion that learning rate warmup helps stabilize the fully connected layers. (3) They also study model distillation with CCA, finding out that the higher layers are the most similar to the teacher model.
Clarity: The paper is clearly written, cites lots of relevant work, and describes the experiments in detail.
Originality: This paper seems original, while the techniques used are established, they conduct thorough experiments on phenomena in deep learning that haven't been studied.

Comments on Significance and Quality:
I liked parts (2), (3) of the paper most, as it seemed like conclusions from these parts were fairly clear:
Figures 4, 5 make the effect of warm restarts in the large batch setting on FC layers clear: the restarts help the layers stabilize better. I really liked the experiment in 4(d), where they tested this hypothesis by freezing the fully connected layers for the duration of the warmup.  It was interesting to see that this had no effect on the remainder of the trajectory. This seemed to be a good demonstration and investigation of the effect of warm restarts, and I appreciate the tests on different architectures in the supplementary material. I'd be curious to see if there's some way to further incorporate this into learning rate schedules.
I also liked Figure 6, exploring Model distillation, which showed that the higher layers of the shallower network were the most affected by the teacher network. The authors cite related work which suggests only training higher layers, and I'd be curious to see how only training higher layers affects accuracy.
While I thought the experiments for part (1) SGD with Restarts were thorough, and appreciated Figure 1, which experimentally validated the use of mode connectivity, I felt there was some difficulty in interpreting the results.
Firstly, in Figure 2, the claim is that SGD with Restarts does possibly bridge local minima as the mode connectivity curves increase between the two convergence points. However, we see in both 2(b) and 2(c) that the linear interpolation between both convergence points does *not* increase in loss. In which case is there any reason to believe that the increase of MC in the middle means that SGDR is climbing a basin? How do we know that the linear combination isn't closer to the path followed by SGDR?
For additional comparisons, it would be good to have the linear combination plots for Figure 1 also.
In general, it seems hard to make meaningful conclusions with low dimensional projections of a very high dimensional loss surface. We'd have to know some kind of theoretical property of MC to be able to do so.
Minor Comments
I think the figures in this paper could be much clearer. In Figure 2 for example, the legend blocks some of the main areas of interest of the plot. I would recommend cutting some of the raw learning rate figures and making all figures much bigger.
In figure 4(d), the text describes the process in training steps (200 training steps), but the plot is in epochs -- it would be better if the text and axis were consistent in units.
Conclusion:
Despite my concerns on the first part of this paper, I think the very thorough experiments, clear presentation and the interesting results on learning rate warmups and model distillation merit its acceptance.


Metric embeddings are immensely useful representations of associations between entities (images, users, search queries, words, and more). Embeddings are learned by optimizing a loss objective of the general form of a sum over example associations. Typically, the optimization uses stochastic gradient updates over minibatches of examples that are arranged independently at random. In this work, we propose principled methods of structuring the arrangements of training examples aimed at accelerating the training. Our arrangements consist of randomized microbatches of examples from a distribution that is guided by the structure of the example associations but respects a specified marginal distribution of training examples. We present efficient microbatch generators and experimentally demonstrate training time acceleration of 3-30%. Structured arrangements emerge as a powerful and novel performance knob for SGD that is independent and complementary to other SGD hyperparameters and thus is a candidate for wide deployment.

The paper presents a method for improving the convergence rate of Stochastic Gradient Descent for learning embeddings by grouping similar training samples together. The basic idea is that gradients computed on a batch of highly associated samples encode related information in a single update that independent samples might take multiple updates to capture. These structured minibatches are constructed by independently combining subsets of positive examples called “microbatches”. Two methods are presented for constructing these microbaches; first by grouping positive examples by shared context (called “basic” microbatches), second by applying Locality Sensitive Hashing to further partition the microbatches into groups that are more likely to contain similar examples.
Three datasets are used for experimental analysis: a synthetic dataset generated using the stochastic block model, and two large scale recommendation datasets. The presented algorithms are compared to a baseline of independently sampled minibatches using the cosine gap and precision for the top k predictions. The authors show the measured cosine gaps over the course of training as well as the gains in training performance for several sets of hyperparameters.
The motivation and basic intuition behind the work is clearly presented in the introductory section. The theoretical justification for the structured minibatches is reasonably convincing and invites empirical verification.
General concerns:
Any method for improving the performance of an optimization process via additional preprocessing must show that the additional overhead incurred from preprocessing the data (in this case, organizing the minibatches) does not negate the achieved improvement in convergence time. This work presents no evidence that this is the case. I expected to see 1) time complexity analysis of each new algorithm proposed for preprocessing and 2) experimental results showing that the overall computation time, including the proposed preprocessing steps, was reduced by this method. Neither of these things are present in this work.
Furthermore, the measured “training gains” are, to my knowledge, not clearly defined. I assume that the authors are using the number of epochs or iterations before convergence as their measure of training performance, but this should be stated explicitly rather than implicitly.
Finally, the experimental results presented do not seem to entirely support the authors’ conclusions. Figures 2, 3, and 4, as well as several of the figures in the appendix, show some parameter settings for which the gains over the baseline are quite limited. This makes me suspect that perhaps the coordinated minibatches aren’t the only variable affecting performance.
I have organized my remaining minor concerns and requests for clarification by section, detailed below.
Section 1
- In the last paragraph, the acronym SGNS is mentioned before being defined. You should either state the full name of the method (with citation) or omit the mention altogether.
Section 2
- I would like a few sentences of additional clarification on what “focus” entities vs. “context” entities are in the more general case. I am familiar with what they mean in the context of Skip Gram, but I think more discussion on how this generalizes is necessary here. Same goes for what kappa (“association strength”) means, especially considering that this concept isn’t really present (to my understanding) in Skip Gram.
- Grammar correction:
“The negative examples provide an antigravity effect that prevents all embeddings to collapse into the same vector”
“to collapse” -> “from collapsing”
Section 3
- Maybe this is just me, but I find the mu-beta notation for the microbatch distributions rather odd. Why not just use a single symbol?
- I would like a bit more clarification on the proof for lemma 3.1, specifically on the last sentence, “the product of these events …”; that statement did not follow obviously to me.
Section 3.1
- Remove the period and colon near kappas at the end of paragraph 3. It’s visually confusing with the dot index notation right next to them.
Section 4
- Typo: “We selects a row vector …” -> “We select a row vector …”
Section 5
- I don’t understand what Figure 1 is trying to demonstrate. It doesn’t do anything (as far as I can tell) to defend the authors’ claim that COO provides a higher expected increase in cosine similarity than IND.
Section 6
- All figures in this section should have axis labels. The captions don’t sufficiently explain what they are.
Section 6.2
- How is kappa computed for the recommendations datasets? This isn’t obvious at all.


SLICED-WASSERSTEIN AUTO-ENCODERS
In this paper we use the geometric properties of the optimal transport (OT) problem and the Wasserstein distances to define a prior distribution for the latent space of an auto-encoder. We introduce Sliced-Wasserstein Auto-Encoders (SWAE), that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or having a likelihood function specified. In short, we regularize the auto-encoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a samplable prior distribution. We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Auto-Encoders (WAE) and Variational Auto-Encoders (VAE), while benefiting from an embarrassingly simple implementation. We provide extensive error analysis for our algorithm, and show its merits on three benchmark datasets.
Learning such generative models boils down to minimizing a dissimilarity measure between the data distribution and the output distribution of the generative model. To this end, and following the work of  and , we approach the problem of generative modeling from the optimal transport point of view. The optimal transport problem Villani (2008); Kolouri et al. (2017) provides a way to measure the distances between probability distributions by transporting (i.e., morphing) one distribution into another. Moreover, and as opposed to the common information theoretic dissimilarity measures (e.g., f -divergences), the p-Wasserstein dissimilarity measures that arise from the optimal transport problem: 1) are true distances, and 2) metrize a weak convergence of probability measures (at least on compact spaces). Wasserstein distances have recently attracted a lot of interest in the learning community Frogner et al. (2015); Gulrajani et al. (2017); ; ; Kolouri et al. (2017) due to their exquisite geometric characteristics Santambrogio (2015). See the supplementary material for an intuitive example showing the benefit of the Wasserstein distance over commonly used f -divergences.
In this paper, we introduce a new type of auto-encoders for generative modeling (Algorithm 1), which we call Sliced-Wasserstein auto-encoders (SWAE), that minimize the sliced-Wasserstein distance between the distribution of the encoded samples and a samplable prior distribution. Our work is most closely related to the recent work by  and more specifically the follow-up work by . However, our approach avoids the need to perform adversarial training in the encoding space and is not restricted to closed-form distributions, while still benefiting from a Wasserstein-like distance measure in the latent space. Calculating the Wasserstein distance can be computationally expensive, but our approach permits a simple numerical solution to the problem. Finally, we note that there has been several concurrent papers, including the work by Deshpande et al. (2018Deshpande et al. ( ) andŞimşekli et al. (2018, that also looked into the application of sliced-Wasserstein distance in generative modeling. Regardless of the concurrent nature of these papers, our work remains novel and is distinguished from these methods. Deshpande et al. (2018) use the sliced-Wasserstein distance to match the distributions of high-dimensional reconstructed images, which require large number of slices, O(10 4 ), while in our method and due to the distribution matching in the latent space we only need O(10) slices. We also note that Deshpande et al. (2018) proposed to learn discriminative slices to mitigate the need for a very large number of random projections that is in essence similar to the adversarial training used in GANs, which contradicts with our goal of not using adversarial training. Simşekli et al. (2018), on the other hand, take an interesting but different approach of parameter-free generative modeling via sliced-Wasserstein flows.

The authors propose a new autoencoding algorithm for the unsupervised generative modeling which they call Sliced Wasserstein Autoencoders (SWAE). SWAE minimizes a reconstruction cost (measured with respect to the non-negative cost function c(x,x') defined for pairs of input images x, x'), regularized by a penalty measuring a discrepancy between the prior distribution over the latent space qz and the push-forward pz of the unknown data distribution through the deterministic encoder. The authors present an extensive theoretical argument supporting the choice of this objective and a number of empirical results performed on MNIST, LSUN bedrooms, and Celeba.
Even though this paper raises several interesting questions, I have several major issues with it:
****
**** 1. Claim around Equation 3 is not proved.
****
All the sections before 2.3 are providing a rather detailed theoretical argument meant to support the choice of the SWAE objective appearing in Eq. 14 of Section 2.3. Here I wand to point out to a mathematical inaccuracy in the authors' discussions, which may render the whole argument questionable. In short, the authors claim around Eq. 3 that "Eq. 3 is equivalent to Theorem 1 of [1] for deterministic encoder-decoder pairs" and don't provide any proofs for this nontrivial fact.
The following is based on some quick derivations I did while reviewing.
Recall that in the current paper Px is the data distribution, Py is the push-forward of Px through the superposition of the encoder \phi and decoder \psi (in other words Py is a distribution of \psi(\phi(X)) when X is distributed according Px). The authors state that:
   \inf_{\phi, \psi} Wc(Px, Py)
    is equivalent to
   (* ) \inf_{\phi, \psi} E_{X \sim Px}[ c(X, \psi(\phi(X))) ].
In other words, the authors state that using Theorem 1 of [1] they are able to show that minimizing a c-optimal transport distance between Px and Py (which is parametrized by \psi and \phi) is *equivalent* to an unconstrained optimization problem appearing on the r.h.s. of Equation 3.
Now, the Theorem 1 of [1] referenced by authors states that if Pz is any prior distribution over the latent space and \psi * Pz is its push-forward through the deterministic decoder \psi, then the optimal transport between Px and the resulting latent variable model \psi * Pz can be equivalently written as:
   (**) Wc(Px, \psi * Pz) = \inf_{f such that f * Px = Pz} E_{X \sim Px}[ c(X, \psi(f(X))) ].
Importantly, note how the right hand side of (**) contains a constrained optimization over an auxiliary (encoder) function f, which does not appear at all in the left hand side. If the authors were to apply (**) directly, they would arrive at the following statement:
   \inf_{\phi, \psi} Wc(Px, Py)
    is equivalent to
   (*** ) \inf_{\phi, \psi} \inf_{f such that f * Px = \phi * Px} E_{X \sim Px}[ c(X, \psi(f(X))) ].
Finally, comparing (*) stated by the authors and (***) obtained above, we see that (*) is obtained by selecting one particular function f = \phi from the set {f such that f * Px = \phi * Px}. Meanwhile, this set in general may contain multiple other functions f and as a result this only shows that (*) >= (***) (as we replace \inf_f with one particular choice of f). However, in this case, I think it is indeed possible to show that (*) = (***). Imagine (***) has a global minimum at (\psi_0,\phi_0, f_0), that is the global optimum of (***) equals E_{X \sim Px}[ c(X, \psi_0(f_0(X))) ]. The same value can be achieved by (*) by setting \phi = f_0. QED.
Once again, these are my preliminary derivations and they need to be checked. But it looks like the claim of the authors is indeed true.
****
**** 2. Empirical evidence is not convincing. ****
****
The main topic of the paper is the unsupervised generative modeling, and the authors claim certain improvements in this field compared to the previous literature. Even though there are no ultimate evaluation metrics available in the field, recently the researchers started supporting their methods with several metrics, including FID scores. By now for most of the widely used datasets the state of art FID scores are well known. In all the experiments the authors provide pictures and interpolations (last row of Fig. 3, Fig. 5) without numbers. I would say nowadays presenting pictures is not enough (being too subjective) and at least some objective numbers (preferably FID) capturing the quality of generated samples should be reported. The authors go into detailed measurements of discrepancy between the aggregate posterior pz and the prior qz, but it is not clear how this affects the actual sample generation. Finally, it is not clear why the authors compare only to WAE-GAN and did not consider WAE-MMD, which is free of adversarial training (in contrast to WAE-GAN) and thus has a stable training and does not involve extra computations of updating the discriminator (as noted by authors on page 10).
[1] Bousquet et al., 2017.


In this paper, we extend the Bayesian neural network to functional Bayesian neural network with functional Monte Carlo methods that use the samples of functionals instead of samples of networks' parameters for inference to overcome the curse of dimensionality for uncertainty quantification. Based on the previous work on Riemannian Langevin dynamics, we propose the stochastic gradient functional Riemannian dynamics for training functional Bayesian neural network. We show the effectiveness and efficiency of our proposed approach with various experiments.

The authors propose an approximate MCMC method for sampling a posterior distribution of weights in a Bayesian neural network.  They claim that existing MCMC methods are limited by poor scaling with dimensionality of the weights, and they propose a method inspired by HMC on finite-dimensional approximations of measures on an infinite-dimensional Hilbert space (Beskos et al, 2011).  In short, the idea is to use a low dimensional approximation to the parameters (i.e. weights) of the neural network, representing them instead as a weighted combination of basis functions in neural network parameter space.  Then the authors propose to use HMC on this lower dimensional representation.  While the idea is intriguing, there are a number of flaws in the presentation, notational inconsistencies, and missing experiments that prohibit acceptance in the current form.
The authors define a functional, f: \theta -> [0, 1], that maps neural network parameters \theta to the unit interval.  They claim that this function defines a probability distribution on \theta, but this not warranted.  First, \theta is a continuous random variable and its probability density need not be bounded above by one; second, the authors have made no constraints on f actually being normalized.
The second flaw is that the authors equate a posterior on f given the data with a posterior on the parameters \theta themselves.  Cf. Eq 4 and paragraph above.  There is a big difference between a posterior on parameters and a posterior on distributions over parameters.   Moreover, Eq. 5 doesn't make sense: there is only one posterior f; there are no samples of the posterior.
The third problem appears in the start of Section 3, where the authors now call the posterior U(theta) instead of f.  They make a finite approximation of posterior U(\theta) = \sum_i \lambda_i u_i, which is inconsistent with Beskos et al.  I believe the authors intend to use a low dimensional approximation to \theta rather than its posterior U(\theta).  For example, if \theta = \sum_i \lambda_i u_i for fixed basis functions u_i, then you can approximate a posterior on \theta with a posterior on \lambda.
The fourth, and most important problem, is that the basis functions u_i are never defined.  How are these chosen? Beskos et al use the eigenfunctions of the Gaussian base measure \pi_0, but no such measure exists here.  Moreover, this choice will have a substantial impact on the approximation quality.
There are more inconsistencies and notational problems throughout the paper.  Section 4.1 begins with a mean field approximation that seems out of place.  Section 3 clearly states that the posterior on theta is approximated with a posterior on lambda, and this cannot factorize over the dimensions of theta.  Finally, the authors again confuse the posterior on weights with a posterior on distributions of weights in Eq 11.   \tilde{U} is introduced as a function of lambda in Eq 14 and then called with f in line 4 of Alg. 1.  These two types are not interchangeable.
These inconsistencies cast doubt on the subsequent experiments.  Assuming the algorithm is correct, a fundamental experiment is still missing.
To justify this approach, the authors should show how the posterior approximation quality varies as a function of the size of the low dimensional approximation, D.
I reiterate that the idea of approximating the posterior distribution over neural network weights with a posterior distribution over a lower dimensional representation of weights is interesting.  Unfortunately, the abundance of errors in presentation cloud the positive contributions of this paper.


Stacked hourglass network has become an important model for Human pose estimation. The estimation of human body posture depends on the global information of the keypoints type and the local information of the keypoints location. The consistent processing of inputs and constraints makes it difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass network. In this paper, we propose a Multi-Scale Stacked Hourglass (MSSH) network to highlight the differentiation capabilities of each Hourglass network for human pose estimation. The pre-processing network forms feature maps of different scales, and dispatch them to various locations of the stack hourglass network, where the small-scale features reach the front of stacked hourglass network, and large-scale features reach the rear of stacked hourglass network. And a new loss function is proposed for multi-scale stacked hourglass network. Different keypoints have different weight coefficients of loss function at different scales, and the keypoints weight coefficients are dynamically adjusted from the top-level hourglass network to the bottom-level hourglass network. Experimental results show that the proposed method is competitive with respect to the comparison algorithm on MPII and LSP datasets.

In this paper a method for pose estimation is proposed, which is based on the well known neural model “stacked hourglass networks”. The novelty in the proposed paper is a multi-scale formulation, which creates multiple scales from the input image and feeds them into different hourglass modules. The different scales are weighted differently, where the weights for a given scale depend on the error obtained on previous scales.
Important issues:
The novelty seems to be not sufficient to me, as multi-scale solutions are not new in computer vision and have been applied a lot in pose estimation as well, be it deep neural models or older techniques. In particular in deep models, multi-scale techniques have been proposed extensively for resolution preserving image to image mappings (which is done here), beginning with quite “old” techniques (in deep learning time scales) going back to 2012 (Farabet, C., Couprie, C., Najman, L., LeCun, Y., 2012. Scene parsing with multiscale feature learning, purity trees, and optimal covers. In: ICML.), or formulations which integrate scales and layers, starting with hyper-columns in 2015 (Hypercolumns for object segmentation and fine-grained localization, CVPR 2015) and many more recent variants.
In 2018, multi-scale formulations are now standard techniques in computer vision with deep networks. I am not sure how the proposed method makes a difference. Also, I am wondering whether there shouldn’t be some parameter sharing between the models of the different scales, as is often done in the literature now to reduce model capacity.
The way the multi-scale Since different resolutions are created is particular. Why not just simply subsample the input images to different resolutions? Why are these trained layers needed as a preprocessing?
Weighting different scales is not fundamentally new. We also don’t know whether it improves performance, it is not part of the ablation study.
The method has been compared to several methods, but which are not state of the art anymore. Most papers are from 2016, the field advanced quickly. The performance gains of the multi-scale formulation are pretty low, and overally, the method is not state of the art on the targeted benchmark. Obviously I do not want to say that a paper needs to be state of the art on a benchmark to get published, even at a top-level conference, far from it. However, if the methodological and theoretical contribution of the paper is rather minor, than the performance and evaluation should be flawless.
Minor remarks:
The writing of this paper is somewhat fuzzy, using non-standard technical language, which I could not decipher and which seems to me somewhat misleading. Examples are:
- “the stacked hourglass network theoretically increases the stacked depth”: how does theory tell us anything about depth in this context?
- “difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass”: I don’t understand what this means, simply. This phrase is also repeated several times in the paper in different places
- “information loss caused by the functional consistency of hourglass networks”: I don’t understand what functional consistency means here, and why it leads to information loss.
The description of Algorithm 1 does not seem to be necessary to me. It basically follows from the equations 4-6, which are executed in order and per layer.


In adversarial attacks to machine-learning classifiers, small perturbations are added to input that is correctly classified. The perturbations yield adversarial examples, which are virtually indistinguishable from the unperturbed input, and yet are misclassified. In standard neural networks used for deep learning, attackers can craft adversarial examples from most input to cause a misclassification of their choice. We introduce a new type of network units, called RBFI units, whose non-linear structure makes them inherently resistant to adversarial attacks. On permutationinvariant MNIST, in absence of adversarial attacks, networks using RBFI units match the performance of networks using sigmoid units, and are slightly below the accuracy of networks with ReLU units. When subjected to adversarial attacks based on projected gradient descent or fast gradient-sign methods, networks with RBFI units retain accuracies above 75%, while ReLU or Sigmoid see their accuracies reduced to below 1%. Further, RBFI networks trained on regular input either exceed or closely match the accuracy of sigmoid and ReLU network trained with the help of adversarial examples. The non-linear structure of RBFI units makes them difficult to train using standard gradient descent. We show that RBFI networks of RBFI units can be efficiently trained to high accuracies using pseudogradients, computed using functions especially crafted to facilitate learning instead of their true derivatives.

This paper introduces a new neural network layer for the purposes of defending against "white-box" adversarial attacks (in which the adversary is provided access to the neural network parameters). The new network unit and its activation function are constructed in such a way that the local gradient is sparse and therefore is difficult to exploit to add adversarial shifts to the input. To train the networks in the presence of a sparse gradient signal, the authors introduce a "pseudogradient", and optimize this proxy-gradient to optimize the parameters. This training procedure shows competitive performance (after training) on the permutation-invariant MNIST dataset versus other more standard network architectures, but is more robust to both adversarial attacks and random noise.
High-level comments:
- Using only a single dataset, and one on which the classification problem is rather easy, is cause for concern. I would need to see performance on another dataset, like CIFAR 10, to be more convinced that this is a general pipeline. In Sec 4, the authors mention that, using the pseudogradient, "one may be concerned that ... we may converge ... and yet, we are not at a minimum of the loss function". They claim that "in practice it does not seem to be a problem" on their experiments. This claim is a bit weak considering only a single, simple dataset was used for training. It is not obvious to me that this would succeed for more complex datasets.
- I would also like to see an additional set of adversarial attacks that are "RBFI-aware". A motivated attacker who is aware of this technique might replace the gradient in the adversarial attack with the pseudogradient instead; I expect such an attack would be effective. While problematic in general, I do not think this is necessarily an overall weakness of the paper (since we, the community, should be investigating methods like these to obfuscate the process of exploiting neural network models), but I would still like to see results showing the impact/performance of adversarial training over the pseudo-gradient. (I do not expect this will be very much effort.)
- What is the purpose of showing robustness of your network models to random noise? It is nice/interesting to see that your results are more robust to random noise, but what is the intuition for why your network performs better?
Wording and minor comments:
- The abstract is rather lengthy, but should probably contain somewhere a spelling-out of RBFI, since it informs the reader that the radial basis function (with infinity-norm) is the structure of the new network unit.
- Sec 4: "...indicate that pseudogradients work much better than regular gradients" :: Please be more clear that this is context specific "...than regular gradients for training RBFI networks".
- Sec. 4 :: Try to be consistent to how you specify "z" in this section, you alternate between the 'infinity-norm' definition and the 'max' definition from Eq. (2). Try to homogenize these.
- In general, the paper was well-proofed and well-written and was easy to read (high clarity).
- To my knowledge, this work is a rather unique foray into solving this problem (original).
Overall, I think this work is an interesting idea to address a rather important concern in the Deep Learning community. While the idea has merit, the small set of experiments in this paper is not sufficiently compelling for me to immediately recommend publication. With a bit more work put into exploring the performance of this method on other datasets, this paper could be made more complete. (Also, since I am aware that space is limited, some of the details on the adversarial attacks from other publications can probably be moved to an appendix.)