Universal feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. A common paradigm is to pre-train a feature extractor on large amounts of data then fine-tune it as part of a deep learning model on some downstream task (i.e. transfer learning). While effective, feature extractors like BERT may be prohibitively large for some deployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.

This work is an empirical study of testing how pruning at the pre-training stage affects subsequent transfer learning (through fine-tuning) stage. The main idea is to carefully control the amount of sparsity injected into BERT through weight magnitude pruning and study the impact on accuracy. The experimental setup is mostly well done, especially the part that disentangles the complexity restriction and information deletion. During the exploration, the authors made several interesting observations, such as 30-40% model weights do not encode any useful inductive bias, which could help shed some light for future work on both training and compressing BERT-like models.
Overall, the paper is well written and explained. The goal is meaningful, and this is a sensible contribution to the ongoing interests of compressing BERT-like large models for efficient training and inference.
My major concern is on its novelty and how directly it can provide benefit to computation.  First, although the findings are interesting, the methods used in this paper are not new.  Various pruning techniques have been explored in prior work, which makes the novelty contribution of this paper somewhat limited.
Furthermore, the study has mostly focused on the impact of random sparsity to accuracy. However, as it is known that it is really difficult for modern hardware to benefit from random sparsity because it leads to irregular memory accesses, which negatively impact the performance. It has been observed that speedups are very limited or can be negative even the random sparsity is >95% [1]. Therefore, it is hard to judge how inference or training can benefit from 30-40% weight sparsity. Going forward, the authors are encouraged to choose pruning methods that lead to regular memory access to avoid adversely impacting practical acceleration in modern hardware platforms.
[1] Learning Structured Sparsity in Deep Neural Networks. Wen et al. NeurIPS 2016


How well can hate speech concept be abstracted in order to inform automatic classification in codeswitched texts by machine learning classifiers? We explore different representations and empirically evaluate their predictiveness using both conventional and deep learning algorithms in identifying hate speech in a ~48k human-annotated dataset that contain mixed languages, a phenomenon common among multilingual speakers. This paper espouses a novel approach to handle this challenge by introducing a hierarchical approach that employs Latent Dirichlet Allocation to generate topic models that feed into another high-level feature set that we acronym PDC. PDC groups similar meaning words in word families during the preprocessing stage for supervised learning models. The high-level PDC features generated are based on Ombui et al, (2019) hate speech annotation framework that is informed by the triangular theory of hate (Stanberg,2003). Results obtained from frequency-based models using the PDC feature on the annotated dataset of ~48k short messages comprising of tweets generated during the 2012 and 2017 Kenyan presidential elections indicate an improvement on classification accuracy in identifying hate speech as compared to the baseline

Comments:
-This paper considers codeswitched hate speech texts from an NLP perspective.  The dataset considers mixed languages.
-Focuses on kenyan presidential election.
-Paper has severe formatting issues as well as simple issues like capitalization.  Additionally many plots are rather unattractive (seem to be produced using Excel or Google Sheets, whereas generally something like matplotlib or seaborn is preferred).
  -The paper puts a lot of examples into the main text, whereas these are usually put into the appendix, or only a few examples are placed in the main text.  Usually the main text focuses more on higher level analysis.
  -Figure 2 is formatted incorrectly (the caption runs on to the next page)
  -I appreciate the effort that went into data annotation as well as the disclosure of the demographics of annotators.
  -Table 1 should make the metric much clearer (it's mentioned in the main text, but it should be in the caption too, also the best performance usually should be bolded)!  Generally TF-idf features or PDC features seem to have the best performance.  The performance of the CNN does not seem very strong.  I think a simple RNN based approach might also be worth considering.  It would also be worth analyzing if the differences between the methods is attributable to underfitting or overfitting.
  -If the paper is proposing a new task with many baselines, it's also important to release the dataset and code in my opinion (I believe ICLR allows this to be done in a de-anonymized way).
Review:
This paper deals with an important problem in social media analysis.  With the spread of hate speech and hate crimes by rioting separatists in Hong Kong as well as equally hateful attacks on Chinese people in the west, I think that this is an issue that deserves more attention in our community.  Unfortunately this paper needs more polish to be appropriate for ICLR.  It also might be better suited to an NLP focused conference (such as ACL, EMNLP, or NAACL) although I think if the technical contribution is clear enough it could be suitable for ICLR as well.
I think the big things to focus on would be including more baselines, improving polish in the paper, and providing a clearer high-level analysis of the dataset (with specific examples mostly left for the appendix).


The recent expansion of Machine Learning applications to molecular biology proved to have a significant contribution to our understanding of biological systems, and genome functioning in particular. Technological advances enabled the collection of large epigenetic datasets, including information about various DNA binding factors (ChIP-Seq) and DNA spatial structure (Hi-C). Several studies have confirmed the correlation between DNA binding factors and Topologically Associating Domains (TADs) in DNA structure. However, the information about physical proximity represented by genomic coordinate was not yet used for the improvement of the prediction models. In this research, we focus on Machine Learning methods for prediction of folding patterns of DNA in a classical model organism Drosophila melanogaster. The paper considers linear models with four types of regularization, Gradient Boosting and Recurrent Neural Networks for the prediction of chromatin folding patterns from epigenetic marks. The bidirectional LSTM RNN model outperformed all the models and gained the best prediction scores. This demonstrates the utilization of complex models and the importance of memory of sequential DNA states for the chromatin folding. We identify informative epigenetic features that lead to the further conclusion of their biological significance.

The authors compared a bidirectional LSTM to traditional machine learning models for genomic contact prediction. Although the paper presents an interesting application of ML, I vote for rejection since i)  the paper is similar to previously published works and lacks methodological novelty, ii) the description about the data and methods is not written clearly enough, and iii) the evaluation needs to be strengthened by additional baseline models and evaluation metrics.
Major comments
=============
1. As pointed out by Maria Anna Rapsomaniki, the paper is similar to previously published papers, which are not cited.
2. It is unclear which features were used as inputs. How were the features that are described in section 3.1 represented? Are these binary values or the mean of the Hi-C signal over the genomic region? What does 20kb mean? How were genomic segments of different chromosomes treated?
3. The motivation of using a biLSTM in section 4.3 is unclear. What has the choice of a biLSTM to do with the fact that DNA is double stranded? The DNA is not used as input to the model and genomic contacts can be formed between non-adjacent segments. Please compare recurrent models to non-recurrent models such as fully connected or convolutional networks.
4. The authors used the weighted MSE as evaluation metric, which was used as training loss of the biLSTM, but it is unclear if the same loss was used for linear and gradient boosting regression. To understand if the performance differences are due to the loss function or model architecture, the authors should use the same loss function for training all models, and use additional evaluation metrics such as the MSE and R^2 score.
5. The authors used linear regression weights to quantify feature importance (figure 4). It is unclear if the biLSTM assigned the same importance to features. To quantify the importance of features learned by the biLSTM, the authors could consider correlating the activations of LSTM units with input features, and also analyze if importances depend on the position of features.
6. Which hyper-parameters of the biLSTM and baseline method did the authors tune and how?
7. The authors should compare the described weighted MSE to the mean squared logarithmic error loss, which is commonly used for penalizing large errors less.
8. Why did the authors use the activation of the center LSTM hidden state instead of concatenating the last hidden state of the forward and reverse LSTM? By using the center hidden state, half of the forward and reverse LSTM activations are ignored.


In this paper, we consider the problem of training structured neural networks (NN) with nonsmooth regularization (e.g. 1 -norm) and constraints (e.g. interval constraints). We formulate training as a constrained nonsmooth nonconvex optimization problem, and propose a convergent proximal-type stochastic gradient descent (Prox-SGD) algorithm. We show that under properly selected learning rates, with probability 1, every limit point of the sequence generated by the proposed Prox-SGD algorithm is a stationary point. Finally, to support the theoretical analysis and demonstrate the flexibility of Prox-SGD, we show by extensive numerical tests how Prox-SGD can be used to train either sparse or binary neural networks through an adequate selection of the regularization function and constraint set.

[Summary]
This paper proposes Prox-SGD, a theoretical framework for stochastic optimization algorithms that (1) incorporates momentum and coordinate-wise scaling as in Adam, and (2) can handle constraint and (non-smooth) regularizers through the proximal operator. With proper choices of hyperparameters, the algorithm is shown to converge asymptotically to stationarity, for smooth non-convex loss + convex constraint/regularizer. The algorithm is empirically tested on training binary and sparse neural nets on MNIST and CIFAR-10.
[Pros]
The theoretical framework that incorporates most of the commonly used tweaks in stochastic optimization for deep learning, and a convergence result that establishes broadly the asymptotic convergence to stationarity.
[Cons]
The result of Theorem 1 sounds rather a straightforward application of classical results; important sub-cases such as Adam violates the assumption (Adam has \rho_t = \rho so \sum \rho_t^2 = \infty) and thus are not contained in this case. From this it seems to me Theorem 1 says things mostly about the “easier” sub-cases, and thus is perhaps not very surprising and a bit limited in bringing in new messages.
The experiments are mostly done on simple problems --- 3 of the 4 figures are on MNIST. The specific tasks (training sparse / binary neural nets with MLP / vanilla CNN architectures) considered in the experiments are all very extensively studied in prior work, and the results in this paper says at most that the proposed Prox-SGD works for these tasks.
Overall, I like the idea in this paper that we can put together a unified framework for stochastic optimization algorithms and incorporate things like momentums and regularizations that were previously treated separately. However, beyond proposing such a framework, it seems that contributions on both the theoretical and empirical side are a bit limited at this point.
***
Thank the authors for the response and the efforts in revising the paper. I am glad to see the additional experiments for training sparse networks on CIFAR-100 (a much harder task than MNIST and also CIFAR-10) in which the proposed method works well. This largely resolved my concerns on the experimental side.
However, I'd still like to hold my evaluation on the theoretical side, in that approaches for handling constraints / non-smoothness / momentums are fairly well understood in the optimization literature. The present result (Theorem 1) conveys the message that these approaches can be combined to work (give an algorithm that converges to stationary points if it converges), but is not really a result that gives us new understandings / novel proof techniques beyond that.
I have slightly improved my rating to reflect my updated evaluation.


This paper proposes a novel per-task routing method for multi-task applications. Multi-task neural networks can learn to transfer knowledge across different tasks by using parameter sharing. However, sharing parameters between unrelated tasks can hurt performance. To address this issue, routing networks can be applied to learn to share each group of parameters with a different subset of tasks to better leverage tasks relatedness. However, this use of routing methods requires to address the challenge of learning the routing jointly with the parameters of a modular multi-task neural network. We propose the Gumbel-Matrix routing, a novel multi-task routing method based on the Gumbel-Softmax, that is designed to learn fine-grained parameter sharing. When applied to the Omniglot benchmark, the proposed method improves the state-of-the-art error rate by 17%.

In many ways this work is well presented. However, I have major concerns regarding the novelty of the proposed method and the theoretical rationale for the key design choices. Although the authors do cite and discuss (Rosenbaum et al., 2019), what is very much not clear to me is how the Gumbel-Matrix Routing proposed in this work differs from past work using the Gumbel Softmax within routing networks. It seems like past work even focused on using only the task for routing, so it is not clear to me how the approach here is really novel in comparison. Even if there is some distinction I am missing, the high level idea is clearly not that new. Additionally, there is not much theoretical discussion about what the Gumbel Softmax adds to routing networks.
The bias/variance tradeoff of Gumbel Softmax / RELAX / REINFORCE was already highlighted in (Rosenbaum et al., 2019). Can the performance of the model on the settings tested be attributed to this tradeoff? If so, would a RELAX model perform even better? Moreover, there is not much discussion of important implications of using the Gumbel Softmax trick in the context of routing. First, as the authors acknowledge, but don't really elaborate on, using the Gumbel Softmax means we must backprop through every possible routing choice in each layer. As a result, the Gumbel approach results in a large scaling of computation with the number of modules, limiting the applicability to more ambitious settings. Moreover, while a clear motivation of this work is eliminating interference between tasks, it is not really explained how Gumbel Softmax does this and how it compares to hard routing decisions in this respect. During backprop, the computation it very similar to mixtures of experts models, and should contain more interference than hard routing. Can you explicitly show that the shape of the Gumbel distribution results in less interference between modules during learning than the standard mixtures of experts softmax approach?
Furthermore, (Rosenbaum et al., 2019) found that a number of RL based models outperform Gumbel Softmax when routing on multi-task settings of CIFAR-100 and the Stanford Corpus of Implicatives. The authors do not provide any explanation for why this approach did not succeed in their settings. This also leads me to doubt how impressive the results presented here are as there is really not any apples to apples comparison with the same architecture and different routing decisions. In Tables 1 and 2 the best baseline is full sharing. This indicates to me that the performance difference with other cited baselines has to do with different architecture choices and not changes in the routing policy itself. The experiments can be much improved by discussing why past approaches to Gumbel based routing have failed and by thoroughly comparing to other methods for just the routing decisions with the same base architecture as done in prior work.  Unfortunately, in its current form, there is not enough context provided for the community to understand the implications of the proposed approach in the submitted draft even though it achieves good performance.


Deep neural networks (DNNs) build high-level intelligence on low-level raw features. Understanding of this high-level intelligence can be enabled by deciphering the concepts they base their decisions on, as human-level thinking. In this paper, we study concept-based explainability for DNNs in a systematic framework. First, we define the notion of completeness, which quantifies how sufficient a particular set of concepts is in explaining a model's prediction behavior. Based on performance and variability motivations, we propose two definitions to quantify completeness. under degenerate conditions, our method is equivalent to Principal Component Analysis. Next, we propose a concept discovery method that considers two additional constraints to encourage the interpretability of the discovered concepts. We use game-theoretic notions to aggregate over sets to define an importance score for each discovered concept, which we call ConceptSHAP. On specifically-designed synthetic datasets and real-world text and image datasets, we validate the effectiveness of our framework in finding concepts that are complete in explaining the decision, and interpretable.

Summary
The paper proposes metrics for evaluating concept based explanations in terms of ‘completeness’ -- characterized by (1) whether the set of presented concepts if sufficient to retain the predictive performance of the original model and (2) how is performance affected when all information useful to a complete set of concepts (as per (1)) is removed from features at a specific layer. Assuming concept vectors lie in linear sub-spaces of the activations of the network at a specific layer, the underlying assumption is that if a given set of ‘concept’ vectors is complete, then using a projection of the intermediate features from input onto the sub-space spanned by concepts should not result in reduced / affected predictive performance. Based on these characterizations, the paper proposes an objective to discovering complete and interpretable set of concepts given a candidate cluster of concepts. Furthermore, the paper proposes metrics to quantify the importance of each concept (using Shapley values) and per-class importance of concepts. The authors conduct experiments on toy data, image and text classification datasets and show that their proposed approach can discover concepts that are complete and interpretable.
Strengths
- The paper is well-written and generally easy to follow. The authors do a good job of motivating the need for the completeness metric, characterizing it specifically in the case of concepts spanning sub-spaces of activations and subsequently utilizing the same to motivate an effective concept discovery method.
- The proposed approach and angle being looked at in the paper is novel in the sense that while prior work has mostly focused on characterizing concepts which are salient. Ensuring that concepts are sufficient for predictive performance ensures the fidelity of the interpretability approach.
- I like the fact that the authors decided to capture both aspects of the completeness criterion -- (1) projection to concept-space should not hurt performance and (2) how does removing concept-projected information from the features affect performance. Capturing both provides a holistic viewpoint of the features of the concerned layer -- (1) can explain features/decisions with an associated metric based on the ‘imperfect’ set of concepts and (2) captures the effectiveness of the information present in the features if we remove all concept-useful information.
- The choice of using Shapley values in ConceptSHAP as a metrics provides a whole range of desirable properties in the metrics used indicate the quality of concepts at different levels of granularity -- per-class importance of concepts across classes adding up to the overall importance of a concept. Furthermore, the observation that under certain conditions, the top-k PCA vectors maximize the defined completeness scores is interesting as well.
Weaknesses
Having said that, I’m interested to hear the thoughts of the authors on the below two points. My primary (not major) concern is the fact the proposed approach is only centered around ensuring and evaluating fidelity of the discovered concepts to the original model.
- While the proposed approach and evaluation metrics are novel, and the results generally support the claims of the paper -- toy experiments result in recovery of the ground-truth concepts, concepts identified for text and image classification offer feasible takeaways -- there is still a lack of proper human-interpretability aspect of the discovered concepts. The proposed approach to discover complete concepts mostly acts similar to a pruning approach on top of a candidate set of concepts based solely on fidelity to the original model. The obtained concepts are mostly explained via feasible hypotheses. One possible experiment that can be used to capture the reliability aspect of the discovered concepts could be as follows -- “Given the set of concepts (and representative patches) and SHAP values across all (or most relevant) classes, are humans able to predict the output of the model?” Is it possible to setup and experiment of this sort? I believe it might help understand the utility of the SHAP values in this context (beyond the advantages in terms of manipulation and characterized completeness).
- The experimental results for image classification are presented on the Animals with Attributes (AwA) dataset. AWA is a fine-grained dataset with only one class present per-image. I’m curious to what happens when the same approach is applied to datasets where images have multiple classes (and potentially distractor classes) present. Is it possible that it becomes harder to discover ‘complete’ concepts (subject to the availability of a decent approach to provide an initial set of candidate clusters). Do the authors have any thoughts on this and any potential experiments that might address this?
Reasons for rating
Beyond the above points of discussion, I don’t have major weaknesses to point out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. The proposed evaluation metrics and discovery objectives offer several advantages and can therefore generally serve as useful quantifiers of concept-based explanation approaches. The strengths and weaknesses highlighted above form the basis of my rating.


It has widely shown that adversarial training (Madry et al., 2018) is effective in defending adversarial attack empirically. However, the theoretical understanding of the difference between the solution of adversarial training and that of standard training is limited. In this paper, we characterize the solution of adversarial training for linear classification problem for a full range of adversarial radius ε. Specifically, we show that if the data themselves are "ε-strongly linearly separable", adversarial training with radius smaller than ε converges to the hard margin solution of SVM with a faster rate than standard training. If the data themselves are not "ε-strongly linearly separable", we show that adversarial training with radius ε is stable to outliers while standard training is not. Moreover, we prove that the classifier returned by adversarial training with a large radius ε has low confidence in each data point. Experiments corroborate our theoretical finding well.

TL;DR: The paper gives interesting, theoretical results to adversarial training. The paper only uses linear classifiers, which are hardly the same problem as deep networks where adversarial attacks are problematic. Some conclusions from theorems can be vague or informal, and therefore are not very convincing. I vote for rejecting this paper since it is hard to claim it informs deep learning research (the motivating reason for doing adversarial training). However, I am not familiar with theoretical analysis of adversarial attack/defense, so I am open to counter-arguments.
=================
1.    What is the specific question/problem tackled by the paper?
The paper gives a theoretical analysis to the theoretically less-studied procedure of adversarial training, and shows properties of adversarial training in comparison to regular training, for both linearly separable data or inseparable data. The paper sheds light on some empirical behavior of adversarially trained networks, namely that they are more robust to outliers and lower in performance.
2.    Is the approach well motivated, including being well-placed in the literature?
I am not an expert of adversarial samples, so I am ill-equipped to judge the novelty of the paper. The research direction itself is well motivated, and in the realm of deep learning, it is posed as a first paper to theoretically analyze adversarial training.
However, the authors only analyzed linear classifiers. This makes the results of the paper ill-suited for deep networks, whose non-linearity is arguably the reason why adversarial samples are such a problem. The motivation of the paper is thus greatly diminished. For linear classifiers, I do not know if there are existing work on their robustness when perturbation of samples are being trained on, but to be well-placed in the literature, the authors must either claim there is none, or cite those papers.
3.    Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
Claims and novelties in this paper include:
(1) Adversarial training converges faster than regular training if samples are ε-strongly linearly separable,
(2) If samples are not "ε-strongly linearly separable", adversarial training is robust to outliers, while regular training is not,
(3) Confidence is low for all (training) samples if ε is large.
Only (1) seems to be sufficiently proved. I am not certain that this is a very useful result, and I am open to counter-arguments.
(2) and (3) have steps that are vague and informal:
(2) That regular training is susceptible to outliers is proof by example and people already know that. Also the claim relies on the assumption that pi^1 and pj^2 are on the same scale, while those samples that violate the decision boundary can have arbitrarily large pi^1 or pj^2. Since outliers are often the violators, and inliers often are not, a small number of carefully placed outliers can make ||p2|| quite large while each pi^1 can be very small. It is also worth noting the logic seems to boil down to "N1>>N2, so inliers should overwhelm outliers, making the training robust". The claim is not guaranteed.
(3) I am not very sure if I understand it correctly, but the logic seems to be that the logits are bounded, so they cannot be too large, and so the confidence is low. However, the bound also involves the magnitude of w and x from the other class, so the final step of the proof is either unclear or the bound can indeed be quite large. Note that |wx| does not need to be very large for the confidence to be high (e.g. if logit is 5, the confidence is 1/(1+e^-5)=99.3%). The claim also relies on the assumption that epsilon is larger than the distance between the farthest points in the dataset, which is extreme since you can find an adversarial sample that can be considered to be simultaneously "close enough" to the two most dissimilar samples in the dataset.
In the end, the results are not very convincing or useful for informing deep learning research.
=============
To improve paper:
- Clarify motivation and how this would inform adversarial training highly non-linear classifiers;
- Add related work for robustness to perturbation of linear models, or state that they don't exist;
- Clarify weaknesses in the claims.
Editorial changes:
Definition 2: "logit" <--- people call this probability estimates; logits are wTx
Sec. 4.1.2 needs to clarify what k in x_i^k means -- it's continued from proposition 1 which at first glance is irrelevant
=================
Post rebuttal
Apologies for not interacting earlier due to deadlines.
The rebuttal does not address my major concern (motivation), nor does it discuss its relationship with related work. The math questions are not answered very clearly; I do not see how section 4.1.2 proves p_i^1 has the same scale as p_j^2, except maybe that they are all smaller than some constant for certain samples. And other discussions in the rebuttal simply confirms my concern.
In summary, I think this paper is not yet ready for publication.


Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes. A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user. To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent. This allows a binary user-level importance weighting of each item without the need to store additional weights for each user. We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG. We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance.

The author proposes a variational encoder (VAE) based approach to perform hashing-based collaborative filtering, which focuses on the weighting problem of each hash-code bit by applying the "self-masking" technique. This proposed technique modifies the encoded information in hash codes and avoids the additional storage requirements and floating point computation, while preserving the efficiency of bit manipulations thanks to hardware-based acceleration. An end-to-end training is achieved by resorting the well-known discrete gradient estimator, straight-through (ST) estimator.
Strength:
The idea of employing the discrete VAE framework to perform hashing-based collaborative filtering is interesting. From the perspective of applications, I think it is somewhat novel.
The experimental results demonstrate the performance superiorities of the self-masking hashing-based collaborative filtering method.
Weakness:
I have concerns over using the VAE to model the ratings of each user and item pairs here. Essentially, you are building a VAE for every rating value R_u, i.  Although the parameters are shared, the VAE’s have their own prior. For generative models like VAE, they need to learn from a lot of data, not just one data point. From the perspective of generating data looking similar to the training data, I don’t think your model have learned anything. Only the reconstruction part is important to your model.
The idea of using discrete VAE for hashing tasks has been explored before, see “NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing”.  It employed a very similar idea, although it is not used for CF, but for hashing directly. The novelties of the model is limited.
Since ratings are essentially ordinal data, using Gaussian distribution to model the rating data may be not appropriate.
The whole paper, especially the model, is not presented well. The model is not presented in a rigorous way, and some sentences in this paper are difficult to follow.
The runtime analysis is not sufficient. In addition to comparing with the methods using hamming distance, we also want to see the advantages of the hamming based method over the real-value based method on speed acceleration.
Other question:
To realize the self-masking role, the paper proposed to use the function f(z_u, z_i)=g(Hamming_self-mask(z_u, z_i)), Equ. 12, and demonstrate the effectiveness of using this function by experiments. Since the necessity of using self-masking is not very convincing, I doubt whether this self-masking function f(z_u, z_i) is indispensable. Maybe, if some other functions that takes z_u and z_i as input are used, better results may be observed.


One of the most fundamental organizational principles of the brain is the separation of excitatory (E) and inhibitory (I) neurons. In addition to their opposing effects on post-synaptic neurons, E and I cells tend to differ in their selectivity and connectivity. Although many such differences have been characterized experimentally, it is not clear why they exist in the first place. We studied this question in deep networks equipped with E and I cells. We found that salient distinctions between E and I neurons emerge across various deep convolutional recurrent networks trained to perform standard object classification tasks, and explored the necessary conditions for the networks to develop distinct selectivity and connectivity across cell types. We found that neurons that project to higher-order areas will have greater stimulus selectivity, regardless of whether they are excitatory or not. Sparser connectivity is required for higher selectivity, but only when the recurrent connections are excitatory. These findings demonstrate that the functional and structural differences observed across E and I neurons are not independent, and can be explained using a smaller number of factors.
Deep neural networks have become powerful tools to model the brain (Yamins & DiCarlo, 2016;Kriegeskorte, 2015). They have been used to successfully model various aspects of the sensory (Yamins et al., 2014;Kell et al., 2018), cognitive (Mante et al., 2013;Wang et al., 2018;Yang et al., 2019), and motor system (Sussillo et al., 2015). Deep networks have been particularly effective at capturing neural representations in higher-order visual areas that remain challenging to model with other methods (Yamins et al., 2014;Khaligh-Razavi & Kriegeskorte, 2014).
Demonstrating that these models can mimic the brain in some respects is only a starting point in the process of advancing neuroscience with deep learning. Once some relationship between artificial and biological networks is established, the model can be dissected to better understand how certain biological computations are mechanically implemented (Sussillo & Barak, 2013). Although notoriously difficult to interpret, deep networks are still much more accessible than the brain itself. This is how deep networks can help address the "how" question. On top of that, deep networks can be used in a normative approach to answer the "why" question. Suppose we found that a certain architecture, objective function (dataset), and training algorithm could together lead to a neural network matching some features of the brain. Then we can ask why the brain evolved such features by testing which element of the architecture, dataset, and training is essential for such features to evolve in the artificial networks. For example, Lindsey et al. (2019) showed early layers of a convolutional neural network can recapitulate the center-surround receptive field observed in retina, but only when followed by an information bottleneck similar to the one that exists from retina to cortex. This finding demonstrates how realistic biological constraints can be used to explain the emergence of known properties of the brain.
Despite the many successes of applying deep networks to model the brain, standard architectures deviate from the brain in many important ways. Recently, neural networks that incorporate fundamental structures of the brain are becoming increasingly common (Kar et al., 2019;Miconi et al., 2018;Song et al., 2016). One fundamental structural principle of the brain is the abundance of recurrent connections within any cortical area. Cortical neurons receive a substantial proportion of their inputs from other neurons in the same area (Harris & Shepherd, 2015). Convolutional recurrent networks trained on object classification tasks can perform comparably to purely feedforward networks with similar numbers of parameters, while being able to better explain temporal responses in higher visual areas (Nayebi et al., 2018;Kar et al., 2019).
One of the most fundamental organizational principles of the cortex is the separation of excitatory (E) and inhibitory (I) neurons (Dale, 1935;Eccles et al., 1954). Dale's law states that each neuron expresses a fixed set of neurotransmitters, which results in either excitatory or inhibitory downstream effects. In addition to the difference in their immediate impacts on post-synaptic neurons, E and I neurons differ in several other important ways. There are several times (4-10x) more E neurons than I neurons (Hendry et al., 1987). Neurons that project to other areas, the so-called "principal neurons", are all excitatory in the cortex (Bear et al., 2007). In the mammalian sensory cortex, E neurons are overall more selective to stimuli than I neurons in the same area as extensively reported in mice (Kerlin et al., 2010;Znamenskiy et al., 2018) and to a lesser extent in other species (Wilson et al., 2017). Finally, E neurons are more sparsely connected with each other, with a connection probability of around 10% (Harris & Shepherd, 2015), compared to I neurons, which target almost all neighboring E neurons (Pfeffer et al., 2013).
Although many of these differences across E and I neurons are well-known and routinely incorporated in computational models (Somers et al., 1995;McLaughlin et al., 2000), it remains unclear whether they can emerge in deep neural networks from training. It is also unknown whether these differences are all independent properties, each contributing to the computation, or if some differences are natural results of the others. To answer these questions, we first trained many variants of deep convolutional recurrent neural networks on classical object classification tasks. Across all variants, three structural principles are built in: Dale's law, the abundance of excitatory neurons, and the role of excitatory units as principal neurons. However, other observed properties of biological E-I circuits are not hardwired, and therefore could only evolve under the pressure of performing the task. We found that functional and structural differences across E and I neurons emerged through training. The development of these characteristics allows us to address the "why" question by removing the built-in differences across E and I neurons and monitoring whether specific functional and structural properties still emerge. Our anonymized code is available at https://anonymous.4open.science/repository/d0ae905f-4171-42b0-94b5-abf03d6414aa.

[Update after rebuttal period]
I would like to thank the authors for the detailed response and for addressing the concerns on such a tight schedule.
Overall my two concerns appear to have been right: 1. The object classification task is not really relevant to elicit the observed behavior and 2. Inhibitory neurons are not essential (at least when training with batch norm).
Do the conclusions of the paper about understanding of E/I cells in the brain still hold? I am not really sure. I would like to urge the authors to carefully assess their conclusions again in light of the new evidence and perhaps rephrase abstract, intro and discussion where necessary.
Having said that, I still like the paper and its approach. I think it provides valuable insights to the field (e.g. neuro-inspired architecture design, concerns regarding training objectives, etc.) and important steps in the right direction and could be a valuable contribution even if the original claims are not entirely substantiated.
[Original review]
This paper seeks to understand why excitatory neurons in cortex are more stimulus selective than inhibitory cells and why excitatory cells are more sparsely connected to each other. The authors train a recurrent neural network model with a number of biological constraints on CIFAR-10. These constraints include Dale's law, an unequal number of E vs. I cells and only excitatory connections between layers (areas) of the network. I would summarize the main findings as follows:
- Excitatory principal neurons are more selective and more sparsely connected to each other than local inhibitory neurons, consistent with biology
- Stimulus selectivity and performance depend only on the number of excitatory cells, but neither on the E/I ratio nor the number of I cells
- High selectivity appears to be a general property of principal cells, but does not depend on the sign of local connections, while sparse connectivity is mainly linked to the excitatory local connections
Strengths:
+ Architectural decisions are well motivated from a neuroscientific perspective
+ Provides a hypothesis why certain connectivity and selectivity patterns emerge in the brain by incorporating biological knowledge into neural network models trained on a specific task
+ Well written and clear, logic development of the arguments
Weaknesses:
- Unclear whether classification task is necessary to elicit the authors' observations
- Interneurons seem unnecessary, raising concerns about relevance of results
- Also trains on ImageNet, but only some results are shown; most from CIFAR
Overall I like the paper from the perspective of a neuroscientist, as it provides a kind of normative account of why things in the brain might look the way they do. My enthusiasm is somewhat limited, however, because I am not convinced the classification task is actually what drives the authors' observations. I am willing to increase my score and argue more strongly for the paper if the authors can address this concern, detailed in the following:
A few observations lead me to believe the classification task the networks are trained may not be important to elicit the observed behavior.
1.) The emergent properties in stimulus selectivity show up almost immediately after training starts according to Fig. 5A+B. Performance, on the other hand, takes a few more epochs to pick up according to Fig. 10.
2.) The connectivity preferences that emerge in Figure 5C appear at already 50 epochs where CIFAR-10 performance is at 50%. To put that into perspective, a linear SVM already achieves 40% on this task, and a one-layer multilayer perceptron with only 100 hidden neurons reaches that performance.
3.) The results of Fig. 11 suggest that the number of inhibitory channels is not important. Did the authors try a stripped down version with only excitatory cells?
I suggest the authors train on CIFAR-10 with shuffled labels on the training set [Zhang et al. 2016; https://arxiv.org/abs/1611.03530]. Although performance on the test set would be at chance (no object recognition capabilities), it would be interesting to see whether the selectivity and connectivity properties still emerge.
Finally, (related to 3.) I wonder whether it makes sense to draw conclusions about differences between E and I cells from a model trained on a task where the number of inhibitory cells seems irrelevant. Wouldn't one need to have at least both types of neurons to be required for the task in order to draw such conclusions?
Minor Comments:
- I sometimes found the nomenclature of the multiple models you tried a bit confusing and hard to follow — especially in parts of the Appendix
- The type of operations (convolution, element-wise) in equation 1, together with the meaning of nonlinearities like \sigma_c are only defined one page after, making that section a bit hard to follow.


In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters. (Video  1 )

This paper proposes an off-policy reinforcement learning method in which the model parameters have been updated using the regression style loss function. Specifically, this method uses two regression update steps: one update value function and another one update policy using weighted regression. To compare the proposed method with others [main comprasion], 6 MuJoCo tasks are used for continuous control and LunarLander-v2 for discrete space.
-- Even though this paper has done a good job in terms of running different experiments, the selection of some of the benchmarks seems arbitrary. For example, for discrete action space, this paper uses LunarLander which is rarely used in any papers so it makes very difficult to draw a conclusion based on these results. Common 49 Atari-2600 games should have been used for comparison. The same thing about experiments in section 5.3 is true too as those tasks are not that well-known.
-- The proposed method doesn't outperform previous off-policy methods on Mujoco task (Table 1). Since the main claim of this paper is a new off-policy method, outperforming the previous off-policy methods is a fair game. The current results are not convincing enough.
-- There are significant overlaps between this paper, "Fitted Q-iteration by Advantage Weighted Regression", "Model-Free Preference-Based Reinforcement Learning ", and "Reinforcement learning by reward-weighted regression for operational space control" which makes the contribution of this paper very incremental.
-- The authors used only 5 seeds to run Mujoco experiments. Given the sensitivity of Mujoco for different starting points, the experiments should have been run at least with 10 different seeds.
Questions:
1) Shouldn't be an importance sampling ratio between \pi and \mu in the equations? starting from eq.5.
2) Does the algorithm optimize the respect to $w$ as well? (eq. 15) if yes, why it is not mentioned in algorithm 1? Plus, since $d(s)$ is uniform dist. (at least this is assumed for implementation), eq. 14,15,and 16 (wherever there is d(s)), those can be simplified, e.g. \hat{V} = \sum(w_i V_i), wouldn't be better just introduced simplified version rather than current ones? (referring to the only equation above section 3.3)
3) Is this the same code used to report results in this paper? if yes, I didn't see any seed assignment in the code?! and what is "action_std" in the code?
There are a couple of recent works in merging on-policy with off-policy updates which you might want to cite them.


Our network decomposes an input image into shape, lighting, and albedo with four disentangled representations: identity, expression, pose, and lighting, which allows expression transfer between different face images.
Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem. In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) learned from limited scan data are often adopted to the reconstruction process. However, methods based on linear parametric models cannot generalize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings. Recent methods aim to learn a nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly. However, the models were only trained on a dataset that is generated from a linear 3DMM. Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications. In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections. A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo. Besides, our proposed model disentangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer. Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared to state-of-the-art methods and is robust to various expression, pose, and lighting conditions.

Overview:
This paper introduces a model for image-based facial 3D reconstruction. The proposed model is an encoder-decoder architecture that is trained in semi-supervised way to map images to sets of vectors representing identity (which encodes albedo and geometry), pose, expression and lighting. The encoder is a standard image CNN, whereas the decoders for geometry and albedo rely on spectral graph CNNs (similar to e.g. COMA, Ranjan’18).
The main contribution of the work with respect to the existing methods is the use of additional loss terms that enable semi-supervised training and learning somewhat more disentangled representations. Authors report quantitative results on MICC Florence, with marginal improvements over the baselines (the choice of the baselines is reasonable).
Decision:
The overall architecture is very similar to existing works such as COMA (Ranjan’18) and (Tran’19), including the specific architecture for geometry decoders, and thus the contributions are primarily in the newly added loss terms.
I also find the promise of “disentangled” representation a bit over-stated, as the albedo and base geometry still seem to be encoded in the same “identity” vector (see related question below).
The numerical improvements seem fairly modest with respect to (Tran’19). In addition, there is no numerical ablation study that would demonstrate the actual utility of the main contributions (such as adversarial loss): there are qualitative results but they are not very convincing.
Thus, the final rating “weak reject”.
Additional comments / typos:
* I am not fully following the argument about sharing identity for albedo and shape on p2: “albedo and face shape are decoded ...”. Would it not be more beneficial to have a fully decoupled representation between the albedo and the facial geometry? I do not see how albedo information would be useful for encoding face geometry and vise-versa.
* Authors claim that one of the main drawbacks e.g. of (Train’19) is the fact that they train on data generated from linear 3DMM. This is indeed the case, but it does not seem like here the authors fully overcome this issue: they do have additional weakly-supervised data, but they still strongly rely on linear 3DMM supervision (p6, “pairwise shape loss”, “adversarial loss”), and do not seem to provide experimental evidence that the model will work without it.
* In particular, the “adversarial training” actually corresponds to learning the distribution of the linear 3DMM. Would it not mean that ultimately the model will be limited to learning only linear ? Could you please elaborate on this?
* p3: “allows ene-to-end … training”
* p3: “framework to exact … representations“.
* p8: “evaluation matric”
Update:
Authors did not provide any response, thus I keep my rating.


We investigate PAC-Bayes bounds for deep neural networks through the closely related IB-Lagrangian objective. We first propose to approximate the IBLagrangian through a second order Taylor expansion of the randomized loss around the minimum. For the case of Gaussian priors and posteriors with fixed means and diagonal covariance, we are able to derive a lower bound to this approximation that corresponds to an "invalid" PAC-Bayes prior (a prior that is training set dependent). Our lower bound depends only on the flatness of the minimum, and the distance between the prior and posterior means. Through a number of experiments our lower bound implies easy and and hard cases where one can or cannot prove generalization even through "cheating". Motivated by this result, we see that for the easy cases, a simple baseline closely matches our lower bound and is sufficient to achieve a non-vacuous PAC-Bayes bound. Crucially the baseline prior is centered on the random deep neural network initialization. This suggests that a good prior mean choice is the main innovation in recent non-vacuous bounds with further optimization of the PAC-Bayes bound through SGD having a complementary role.

The authors  replace the empirical risk term in a PAC-Bayes bound by its second-order Taylor series approximation, obtaining an approximate (?)  PAC-Bayes bound that depends on the Hessian. Note that the bound is likely overoptimistic unless the minimum is quadratic. They purpose to study SGD by centering the posterior at the weights learned by SGD. The posterior variance that minimizes this approximate PAC-Bayes bound can then be found analytically. They also solve for the optimal prior variance (assuming diagonal Gaussian priors/posteriors), producing a hypothetical "best possible bound" (at least under the particular choices of priors/posteriors, and under this approximation of the empirical risk term). The authors evaluate their approximate bound and "best bound possible" empirically on MNIST and CIFAR. This requires  computing approximations of the Hessian for small fully connected neural networks trained on MNIST and CIFAR10. There are some nice visualizations (indeed, these may be one of the most interesting contributions.)
The direction taken by the authors is potentially interesting. However, there are a few issues that would have to be addressed carefully for me to recommend acceptance. First, the comparison to (some very) related work is insufficient, and so the actual novelty is misrepresented (see detailed comments below). Further, the paper is full of questionable vague claims and miscites/attributes other work. At the moment, I think the paper is below the acceptance threshold: the authors need to read and understand (!) related work, and expand their theoretical and/or empirical results to produce a contribution of sufficient novelty/impact.
DETAILED FEEDBACK.
I believe the authors missed some related work by Tsuzuki, Sato and Sugiyama (2019), where a PAC-Bayes bound was derived in terms of the Hessian, via a second-order approximation. How are the results presented in this submission relate to Tsuzuki et al approach?
When the posterior, Q, is a Gaussian (or any other symmetric distribution), \eta^T H \eta is the so-called Skilling-Hutchinson trace estimator. Thus E(\eta^T H \eta) is the Trace(H) scaled by the variance of \eta. The authors seem to have completely missed this connection, which simplifies the final expression considerably.
Why is the assumption that the higher order terms are negligible reasonable? Citation or experiments required.
Regarding the off-diagonal Hessian approximation: how does the proposed layer-wise approximation relate to k-FAC (Martens and Grosse 2015)?
IB Lagrangian: I am not sure why the authors state the result in Thm 4.2 as a lower bound on the IB Lagrangian. What’s the significance of having a lower bound on IB Lagrangian?
Other comments:
Introduction: “At the same time neither the non-convex optimization problem solved in .. nor the compression schemes employed in … are guaranteed to converge to a global minimum.”. This is true but it is really not clear what the point being made is. Essentially, so what? Note that PAC-Bayes bounds hold for all posteriors, even ones not centered at the global minimum (of any objective). The claims made in the rest of the paragraph are also questionable and their purposes are equally unclear. I would be grateful if the authors could clarify.
First sentence of Section 3.1: “As the analytical solution for the KL term in 1 obviously underestimates the noise robustness of the deep neural network around the minimum...”. I have no idea what is being claimed here. The statement needs to be made much less vague.  Please explain.
Section 4: “..while we will be minimizing an upper bound on our objective we will be referring with a slight abuse of terminology to our results as a lower bound.”. I would appreciate if the authors could clarify what they mean here.
Section 4.1 beginning: “We make the following model  assumptions...”. Choosing a Gaussian prior and posterior is not an assumption. It's simply a choice. The PAC-Bayes bound is valid for any choices of Gibbs classifiers. On the other hand, it is an assumption that such distributions will yield "tight" bounds, related to the work of Alquier et al.
Section 4.1 “In practice we perform a grid search over the parameters..”. The authors should mention that such a search should be accounted for via a union bound (or otherwise). The "cost" of such a union bound should be discussed.
The empirical risk of Q is computed using 5 MCMC samples. This seems like a very low number, as it would not even give you one decimal point of accuracy with reasonable confidence! The authors should either use more samples, or account for the error in the upper bound using a confidence interval derived from a Chernoff bound.
Section 4.2: “The concept of a valid prior has been formalized under the differential privacy setting...”. I am not sure what the authors mean by that.
Section 5: “There is ambiguity about the size of the Hessians that can be computed exactly.” What kind of ambiguity?
Same paragraph in Section 5 discusses why there are few articles on Hessian computation. The authors claim that “the main problem seems to be that the relevant computations are not well supported...”. This is followed by another comment that is supposed to contrast the previous claim, saying that storing the Hessian is infeasible due to memory requirements. I am not sure how this claim about memory requirements shows a contrast with the claim on computation not being supported.
First sentence in Section 5.1: I believe this is only true under some conditions.
Section 5.1: The authors should explain why they add a damping term, alpha, to the Hessian, and how alpha affects the results.
***
Additional citation issues:
The connections between variational inference, PAC-Bayes and IB Lagrangian have been pointed out in previous work (e.g. Germain, Bach, Lacoste, Lacoste-Julien (2016); Achille and Soatto 2017).
In the introduction, the authors say “...have been motivated simply by empirical correlations with generalization error; an argument which has been criticized …” (followed by a few citations). Note, that this was first criticized in Dziugaite and Roy (2017).
“Both objectives in … are however difficult to optimize for anything but small scale experiments.”. It seems peculiar to highlight this, since the approach that the authors are presenting is actually more computationally demanding.
Citations for MNIST and CIFAR10 are missing.
***
Minor:
Theorem 3.1 “For any data distribution over..”, I think it was meant to be \mathcal{X} \times  (and not \in )
Theorem 4.2: “For our choice of Gaussian prior and posterior, the following is a lower bound on the IB-Lagrangian under any Gaussian prior covariance”. I assume only the mean of the Gaussian prior is fixed.
Citations are misplaced (breaking the sentences, unclear when the paper of the authors are cited).
There are many (!) missing commas, which makes some sentences hard to follow.
***
Positive feedback: I thought the visualizations in Figure 2 and 3 were quite nice.


Recent advances have made it possible to create deep complex-valued neural networks. Despite this progress, the potential power of fully complex intermediate computations and representations has not yet been explored for many challenging learning problems. Building on recent advances, we propose a novel mechanism for extracting signals in the frequency domain. As a case study, we perform audio source separation in the Fourier domain. Our extraction mechanism could be regarded as a local ensembling method that combines a complex-valued convolutional version of Feature-Wise Linear Modulation (FiLM) and a signal averaging operation. We also introduce a new explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram. Using the Wall Street Journal Dataset, we compare our phaseaware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss. When operating in the complex-valued frequency domain, our deep complex-valued network substantially outperforms its real-valued counterparts even with half the depth and a third of the parameters. Our proposed mechanism improves significantly deep complex-valued networks' performance and we demonstrate the usefulness of its regularizing effect.

This paper proposes a new method for source separation, by using deep learning UNets, complex-valued representations and the Fourier domain. Concretely, their contribution is : i) a complex-valued convolutional version of the Feature-Wise Linear Modulation, able to optimise the parameters needed to create multiple separated candidates for each of signal sources that are then combined using signal averaging; ii) the design of a loss that takes into account magnitude and phase while being scale and time invariant. It was then tested and compared with real-valued versions, and also some state-of-the-art methods.
Overall, I think this paper is of good quality and proposes an interesting method for this crucial task of source separation. However, I found the paper too dense and difficult to read (even if well written), and it looks like a re-submission from a journal paper of more than 8 pages. I would suggest the authors to shrink the paper so it *really* fits into the 8-pages (without important figures or important implementation details in the appendices), maybe at the cost of leaving some parts (such as old related works) out of the paper. The experiments are important here, and it is too bad that the comparison with state-of-the-art is just in the last paragraph, while the results do not seem to show any improvements compared to the other methods. The computational time might be very important here, as the claim is that FFT reduces time computation, but I did not had time to go through all the appendices.
Positive aspects:
- The work is well documented and motivated, and I found that the reflexion leading to the method is of good quality.
- Concretely, I found interesting the use of the FiLM, originally designed for another application, for minimizing the SNR of the signal sources. The motivation/proof is quite clear too.
- Equally, the motivation for the design of the new loss is clear and interesting.
- I also found important the experiments, that shows in the same table the difference between the method without the complex-valued part and with different parameter values.
Questions and remarks:
- I have to recall that I am not an expert on source separation and complex-valued deep learning. Yet, I have had difficulties in understanding the structure of the method, even if the different parts were clearly explained. The figure 1 is very useful, but I found it not clear enough and too small. I went to find some informations in the appendices, but there are too much crucial information there and I did not have time to go through all of it.
- The use of the U-net architecture is not explained (just some citations are given). What is supposed to be the output of it?
- When you say 'to be more rigorous, we can assume in the cse of speech separation that, for each speaker, there exists an impulse response such that when it is convolved with the clean speech of the speaker, it allows to reconstruct the mix' : why can we be sure that it is always possible, and why is it more rigourous?
- Why are the additive noises epsilon_i supposed to have the same E(|epsilon_i|^2|) ? Even if they are uncorrelated, what is the hypothesis behind that?
- In the CSimLoss, why (i) is the real part negative and the imaginary part positive; (ii) is the imaginary part squared?
- have you tested with a higher lambda_imag (as the larger is now the best)?
- It looks from the end of the paper that the method is still not achieving better results than the state-of-the-art. I agree with the authors as it might not be the scope of the paper, but then what is it? If it's time computation, it is not shown in the paper. If it is just a methodology, what would be required in the future to beat the best method?
- In the results, table 1, in the last 4 lines: it looks from 1st and 2nd line that the new loss CSimLoss is not very different from the L2 (9.88 compared to 9.87). The best result, in the 4th line, cannot be compared to the 3rd line as both the loss and the number of transforms are different. I then found those values in the appendices, but it would be best to show fewer parameters varying in the main paper, but show some results that can be easily compared.
- What is the importance of the first paragraph in 2.1? I was not aware of the holographic reduced representations, but I don't understand it more now, and I don't see why explaining that for 15 lines.
Small remarks:
- 'deep complex valued models have *just* started to gain momentum'... with citations beginning in 2014, I would not say 'just'.
- 'in the frequncy domain is then, ...' --> frequency + no coma
- Figure 2 is in the appendix, while in the text it is not said so. I was lost. This figure should not be in the appendix as the appendix should not have key elements, but just details that are not important for the understanding of the paper.


Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.

This work proposes to leverage a pre-trained semantic segmentation network to learn semantically adaptive filters for self-supervised monocular depth estimation. Additionally, a simple two-stage training heuristic is proposed to improve depth estimation performance for dynamic objects that move in a way that induces small apparent motion and thus are projected to infinite depth values when used in an SfM-based supervision framework. Experimental results are shown on the KITTI benchmark, where the approach improves upon the state-of-the-art.
Overview:
+ Good results
+ Doesn't require semantic segmentation ground truth in the monodepth training set
- Not clear if semantic segmentation is needed
- Specific to street scenes
- Experiments only on KITTI
The qualitative results look great and the experiments show that semantic guidance improves quantitative performance by a non-trivial factor. The qualitative results suggest that the results produced with semantic guidance are sharper and more detailed. However, it is not clear that using features from a pre-trained semantic segmentation network is necessary. The proposed technical approach is to use the pixel-adaptive convolutions by Su et. al. to learn content-adaptive filters that are conditioned on the features of the pre-trained semantic segmentation network. These filters could in principle be directly learned from the input images, without needing to first train a semantic segmentation network. The original work by Su et. al. achieved higher detail compared to their baseline by just training the guidance network jointly.  Alternatively, the guidance network could in principle be pre-trained for any other task. The main advantage of the proposed scheme is that the guidance path doesn't need to be trained together with the depth network. On the other hand, unless shown otherwise, we have to assume that the network needs to be pre-trained on some data that is sufficiently close to the indented application domain. This would limit the approach to situations where a reasonable pre-trained semantic segmentation network is available.
The proposed heuristic to filter some dynamic objects is very specific to street scenes and to some degree even to the KITTI dataset. It requires a dominant ground plane and is only able to detect a small subset of dynamic motion (e.g. apparent motion close to zero and object below the horizon). It is also not clear what the actual impact of this procedure is. Section 5.4.2 mentions that Abs. Rel decreases from 0.121 to 0.119, but it is not clear to what this needs to be compared to as there is no baseline in any of the other tables with an Abs. Rel of 0.121. Additionally, while the authors call this a minor decrease, the order of magnitude is comparable to the decrease in error that this method shows over the state-of-the-art (which the authors call statistically significant) and also over the baselines (c.f. Table 2). Can the authors clarify this?
Related to being specific to street scenes: The paper shows experiments only on the KITTI dataset. The apparent requirement to have a reasonable semantic segmentation model available, make it important to evaluate also in other settings (for example on an indoor dataset like NYU) to show that the approach works beyond street scenes (which is one of the in practice not so interesting settings for monocular depth estimation since it is rather easy to just equip cars with additional cameras to solve the depth estimation problem).
Need for a reasonable segmentation model: It is not clear in how far the quality of the segmentation network impacts the quality for the depth estimation task. What about the domain shift where the segmentation model doesn't do so well? Even if the segmentation result is not used directly, the features will still shift. How much would depth performance suffer?
Summary:
While the results look good on a single dataset, I have doubts both about the generality of the proposed approach as well as the need for the specific technical contribution.
=== Post rebuttal update ===
The authors have addressed many of my initial concerns and provided valuable additional experimental evaluations. While I'd like to upgrade my recommendation to weak accept, I strongly encourage the authors to provide additional experiments on different datasets (at least NYU).


The ability of graph neural networks (GNNs) for distinguishing nodes in graphs has been recently characterized in terms of the Weisfeiler-Lehman (WL) test for checking graph isomorphism. This characterization, however, does not settle the issue of which Boolean node classifiers (i.e., functions classifying nodes in graphs as true or false) can be expressed by GNNs. We tackle this problem by focusing on Boolean classifiers expressible as formulas in the logic FOC 2 , a well-studied fragment of first order logic. FOC 2 is tightly related to the WL test, and hence to GNNs. We start by studying a popular class of GNNs, which we call AC-GNNs, in which the features of each node in the graph are updated, in successive layers, only in terms of the features of its neighbors. We show that this class of GNNs is too weak to capture all FOC 2 classifiers, and provide a syntactic characterization of the largest subclass of FOC 2 classifiers that can be captured by AC-GNNs. This subclass coincides with a logic heavily used by the knowledge representation community. We then look at what needs to be added to AC-GNNs for capturing all FOC 2 classifiers. We show that it suffices to add readout functions, which allow to update the features of a node not only in terms of its neighbors, but also in terms of a global attribute vector. We call GNNs of this kind ACR-GNNs. We experimentally validate our findings showing that, on synthetic data conforming to FOC 2 formulas, AC-GNNs struggle to fit the training data while ACR-GNNs can generalize even to graphs of sizes not seen during training.

The paper utilizes recent insights into the relationship between the Weisfeiler-Lehman (WL) test for checking graph isomorphism and Graph Neural Networks (GNNs) in order to characterize the class of node classifiers that can be captured by a specific GNN architecture, called aggregate-combine GNN (AC-GNN).
The primary contribution of this work is the identification of the logical classifiers that can be represented within an AC-GNN, a fragment of first order logic called graded modal logic, as well as an extention of AC-GNNs with the ability to capture a strictly more expressive fragment of first order logic, called ACR-GNN. Both of these results are supported by formal proofs and derivations, adding not only theoretical value to the presented work, but also intuitive insights behind the reasons of the AC-GNN limitations. In addition, the presented experiments demonstrate the practical implications of the results, with the exception of some datasets where no significance difference in performance between AC-GNNs and ACR-GNNs was found.
I believe that the contributions of this paper are significant. On the one hand, studying theoretical properties of GNNs facilitates their transparency and highlights the range of applications they can be used in. On the other hand, although the GNN variant introduced by the author is a special case of an existing class of GNNs, the motivation behind its introduction is different and follows naturally from the discussion within the paper.
The fact that no actual difference in performance between AC-GNNs and ACR-GNNs was noticed in the only non-synthetic dataset used in the experiment should prompt the author to run experiments with more real life datasets, in order to empirically verify the results, but this is a minor point.


Pooling operations have shown to be effective on various tasks in computer vision and natural language processing. One challenge of performing pooling operations on graph data is the lack of locality that is not well-defined on graphs. Previous studies used global ranking methods to sample some of the important nodes, but most of them are not able to incorporate graph topology information in computing ranking scores. In this work, we propose the topology-aware pooling (TAP) layer that uses attention operators to generate ranking scores for each node by attending each node to its neighboring nodes. The ranking scores are generated locally while the selection is performed globally, which enables the pooling operation to consider topology information. To encourage better graph connectivity in the sampled graph, we propose to add a graph connectivity term to the computation of ranking scores in the TAP layer. Based on our TAP layer, we develop a network on graph data, known as the topology-aware pooling network. Experimental results on graph classification tasks demonstrate that our methods achieve consistently better performance than previous models.

This paper presented a new pooling method for learning graph-level embeddings. The key idea is to use the initial node attributes to compute all-pair attention scores for each node pair and then use these attention scores to formulate a new graph adjacency matrix beyond the original raw graph adjacency matrix. As the result, each node can average these attention score edges to compute the overall importance. Based on these scores, the method chooses top-k nodes to perform graph coarsening operation. In addition, a graph connectivity term is proposed to address the problems of isolated nodes. Experiments are performed to validate the effectiveness of the proposed method.
Although I found this paper is generally well written and well motivated, there are several concerns about the novelty of paper, computational expenses, and the experimental results as listed below:
1) The idea of using attention score is simple yet seems effective. But using attention score to identify the importance of nodes has been first presented in GAT [Velicˇkovic ́ et al., 2019] and further studied by a lot of subsequent works.
2) In this paper, authors considered using raw node features to obtain good attention scores between each node pair. However, the effectiveness of this will heavily depend on how informative these original node features are. In the extreme case, the original node features are completely random, then the proposed method will definitely fail since there are no meaningful similarity scores that can be learnt from. Surprisingly, I did not see any discussion or any ablation study on this aspect. Also, why only using initial node attributes? How about the computed node embeddings to compute the similarity scores?
3) The pair-wise similarity scores (similarity or attention matrix) are very expensive to compute and score, which easily renders quadratic complexity in terms of the number of nodes O(N^2) for both computation and memory. This makes it scale to really large graph. In this sense, the proposed scheme is not promising in real applications.
4) The experiments are really problematic. There are no descriptions on how the graph data are split in train/val/test. Following the traditional graph kernel settings, it should be 9/1/1. Also, there are no information about how many runs are performed on each dataset. I noticed that authors basically collected all performance results from published related works except for several baselines. Different data split and number of runs could result quite different performance report, leading to apple-and-orange comparisons.
5) The experimental results are also problematic as well. For example, WL baseline has extremely large variance for all of datasets, which are completely different from the reported number from other literature [Zhang et al, neurIPS'18] and my personal experience. In general, the variance of WL could be 1/10 smaller than what are reported in this paper. Also, there is no clear explanations why the proposed method achieved quite large margin compared to G-UNet and GIN as shown in Table 1.
RetGK: Graph Kernels based on Return Probabilities of Random Walks, [Zhang et al, neurIPS'18]
6) In Table 3, authors tried to show the performance difference between different pooling methods. I was so surprised about the results after I looked back to Table 1. For instance, Net_top-k (G-Unet) achieved 71.5 on PTC while GUnet achieved 64.7 on PTC in Table 2. It is really hard to believe this almost 7 points difference are just due to slight different model choices in other parts beyond the main contribution - new pooling component. I feel the whole experimental results are not convincing or at least no well explanations what's going on here. Of course, based on very large variances shown in table 1 and 2, table 3 and table 4 are meaningless to look without seeing the variances for each ablation study.


The contrastive divergence algorithm is a popular approach to training energybased latent variable models, which has been widely used in many machine learning models such as the restricted Boltzmann machines and deep belief nets. Despite its empirical success, the contrastive divergence algorithm is also known to have biases that severely affect its convergence. In this article we propose an unbiased version of the contrastive divergence algorithm that completely removes its bias in stochastic gradient methods, based on recent advances on unbiased Markov chain Monte Carlo methods. Rigorous theoretical analysis is developed to justify the proposed algorithm, and numerical experiments show that it significantly improves the existing method. Our findings suggest that the unbiased contrastive divergence algorithm is a promising approach to training general energy-based latent variable models.

The paper proposes an algorithmic improvement that significantly simplifies training of energy-based models, such as the Restricted Boltzmann Machine. The key issue in training such models is computing the gradient of the log partition function, which can be framed as computing the expected value of f(x) = dE(x; theta) / d theta over the model distribution p(x). The canonical algorithm for this problem is Contrastive Divergence which approximates x ~ p(x) with k steps of Gibbs sampling, resulting in biased gradients. In this paper, the authors apply the recently introduced unbiased MCMC framework of Jacob et al. to completely remove the bias. The key idea is to (1) rewrite the expectation as a limit of a telescopic sum: E f(x_0) + \sum_t E f(x_t) - E f(x_{t-1}); (2) run two coupled MCMC chains, one for the “positive” part of the telescopic sum and one for the “negative” part until they converge. After convergence, all remaining terms of the sum are zero and we can stop iterating. However, the number of time steps until convergence is now random.
Other contributions of the paper are:
1. Proof that Bernoulli RBMs and other models satisfying certain conditions have finite expected number of steps and finite variance of the unbiased gradient estimator.
2. A shared random variables method for the coupled Gibbs chains that should result in faster convergence of the chains.
3. Verification of the proposed method on two synthetic datasets and a subset of MNIST, demonstrating more stable training compared to contrastive divergence and persistent contrastive divergence.
I am very excited about this paper and strongly support its acceptance, since the proposed method should revitalize research in energy-based models. While I find the experiments to be somewhat lacking, this is sufficiently offset by the theoretical contributions of the paper.
Pros
1. The paper reads well and introduces all the necessary preliminaries to understand the method. This is important, since I expect many readers to be unfamiliar with the technique.
2. The proposed method solves an important problem which, as far as I understand, has been the roadblock in large-scale training of RBMs and related models. It is also elegant and fairly straightforward to implement.
3. The proof of finite computation time and variance is very nice to have. This is because in some cases removing the bias leads to infinite variance, e.g. a parallel submission on SUMO (https://openreview.net/forum?id=SylkYeHtwr).
Cons
1. I don’t think Corollary 1 (convergence of gradient descent to the global optimum) is true for RBMs, as stated on Page 6. This is because the log-likelihood of RBM, or indeed any latent-variable model with permutation-invariant latents, is non-convex. I would suggest removing this corollary and simplifying Algorithm 2 to be regular SGD, as used in the experiments.
2. There is no experimental comparison of Algorithm 1 (the general version) and Algorithm 3 (the specialized RBM version). It seems intuitive that the specialized version should have lower computation time, but this must be confirmed.
3. The experimental section may be significantly improved.
* It is unclear what value of k (number of initial Gibbs steps) from Algorithm 2 is used.
* The experiments on just the “0” digits of MNIST seem a bit simplistic for the year 2019. It is also not clear what binarization protocol is used.
* It would be very helpful to provide estimates of the gradient (not log-likelihood) variance of each method to better understand the trade-off between the bias and the variance.
* I would also like to see the wall-clock time comparison of the methods.
Minor comments
* Page 1. Of this kind -> of this class. The data distribution p_v (v; theta) -> The model distribution
* Page 2. Property -> properties. CD-\tau -- I don’t think you can correctly refer to your method in this way, since it has at least double the computation time of CD for the same number of iterations.
* Page 3. Provides -> provide. Likelihood gradient -> log-likelihood gradient
* Algorithm 1 is an infinite loop with no break clause. It would be good to add a break statement after line 5. This would also simplify the discussion of the method.
* Page 7. I wouldn’t call the fact that CD doesn’t converge on the BAS dataset remarkable, given that it’s been reported by Fischer & Igel 2014.
* Page 9. The last paragraph stating that the proposed method is not a replacement for CD is confusing. Can you add a short experiment to demonstrate that this combination makes sense?


While there has been great interest on generating imperceptible adversarial examples in continuous data domain (e.g. image and audio) to explore the model vulnerabilities, generating adversarial text in the discrete domain is still challenging. The main contribution of this paper is to propose a general targeted attack framework AdvCodec for adversarial text generation which addresses the challenge of discrete input space and is easily adapted to general natural language processing (NLP) tasks. In particular, we propose a tree based autoencoder to encode discrete text data into continuous vector space, upon which we optimize the adversarial perturbation. A tree based decoder is then applied to ensure the grammar correctness of the generated text. It also enables the flexibility of making manipulations on different levels of text, such as sentence (AdvCodec(Sent)) and word (AdvCodec(Word)) levels. We consider multiple attacking scenarios, including appending an adversarial sentence or adding unnoticeable words to a given paragraph, to achieve arbitrary targeted attack. To demonstrate the effectiveness of the proposed method, we consider two most representative NLP tasks: sentiment analysis and question answering (QA). Extensive experimental results and human studies show that AdvCodec generated adversarial text can successfully attack the neural models without misleading the human. In particular, our attack causes a BERT-based sentiment classifier accuracy to drop from 0.703 to 0.006, and a BERT-based QA model's F1 score to drop from 88.62 to 33.21 (with best targeted attack F1 score as 46.54). Furthermore, we show that the white-box generated adversarial texts can transfer across other black-box models, shedding light on an effective way to examine the robustness of existing NLP models.

Motivated by recent development of attack/defense methods addressing the vulnerability of deep CNN classifiers for images, this paper proposes an attack framework for adversarial text generation, in which an autoencoder is employed to map discrete text to a high-dimensional continuous latent space, standard iterative optimization based attack method is performed in the continuous latent space to generate adversarial latent embeddings, and a decoder generates adversarial text from the adversarial embeddings.  Different generation strategies of perturbing latent embeddings at sentence level or masked word level are both explored. Adversarial text generation can take either a form of appending an adversarial sentence or a form of scattering adversarial words into different specified positions. Experiments on both sentiment classification and question answering show that the proposed attack framework outperforms some baselines. Human evaluations are also conducted.
Pros:
This paper is well-written overall. Extensive experiments are performed.
Many human studies comparing different adversarial text generation strategies and evaluating adversarial text for sentiment classification/question answering are conducted.
Cons:
1) Although the studied problem in this paper is interesting, the technical innovation is very limited. All the techniques are standard or known.
2) There are two major issues: lacking a rigorous metric of human unnoticeability and lacking justification of the advantage of the tree-based autoencoder. I think the first issue is a major problem that renders all the claims in this paper questionable. The metrics used to define adversarial images for deep CNN classifiers are indeed valid and produce unnoticeable images for human observers. But in this paper, the adversarial attack is performed in the latent embedding space, and there is no explicit constraint enforced on the output text. It’s unconvincing that this approach will generate adversarial text that seems negligible to humans. Therefore, the studied problem in this paper has a completely different nature from the one for CNN image classifiers and it is hard to convince readers that the proposed  framework generates adversarial text legitimate to human readers.
3) It is unclear why tree-structured LSTM instead of a standard LSTM/GRU should be chosen in this framework for adversarial text generation. If this architecture is preferred, sufficient ablation studies should be conducted.
4) In section 3.3, the description about adversarial attacks at word level is unclear. More detailed loss function and algorithms along with equations should be provided.
5) In section 5.2, it is unclear that the majority answers on the adversarial text will, respectively, match the majority answers on the original text. Moreover, it seems that there is a large performance drop from original text to adversarial text. Therefore, it is valid to argue that whether the proposed framework can generate legitimate adversarial text to human readers or not.
6) It’s better to include many examples of generated adversarial text in the appendix.
7) Missing training details: It is unclear how the model architectures are chosen, and learning rate, optimizer, training epochs etc. are also missing. All these training details should be included in the appendix.
8) Minor: Figure 1: "Append an initial sentence...",  section 3: "map discrete text into a high dimensional...",  section 3.2.2: "Different from attacking sentiment analysis..." ....
In summary, the research direction of adversarial text generation studied in this paper is interesting and promising. However, some technical details are questionable, and the produced results without rigorous metrics seem to be unconvincing.


We discuss the approximation of the value function for infinite-horizon discounted Markov Reward Processes (MRP) with nonlinear functions trained with the Temporal-Difference (TD) learning algorithm. We consider this problem under a certain scaling of the approximating function, leading to a regime called lazy training. In this regime the parameters of the model vary only slightly during the learning process, a feature that has recently been observed in the training of neural networks, where the scaling we study arises naturally, implicit in the initialization of their parameters. Both in the under-and over-parametrized frameworks, we prove exponential convergence to local, respectively global minimizers of the above algorithm in the lazy training regime. We then give examples of such convergence results in the case of models that diverge if trained with non-lazy TD learning, and in the case of neural networks.

The paper discusses the policy evaluation problem using temporal-difference (TD) learning with nonlinear function approximation. The authors show that in the “lazy training” regime both over- and under-parametrized approximators converge exponentially fast, the former to the global minimum of the projected TD error and the latter to a local minimizer of the same error surface. Simply put, the lazy regime refers to the approximator behaving as if it had been linearized around its initialization point. This can happen if the approximation is rescaled, but can also occur as a side-effect of its initialization. The authors present simple numerical examples illustrating their claims.
Although I did not carefully check the math, this seems like a solid contribution on the technical side. My main concern about the paper is that it falls short in providing intuition and contextualizing its technical content. Regarding the presentation, I believe it is possible to have a less dry prose without sacrificing mathematical rigor. If some of the technical material is moved to the appendix --like auxiliary results, discussion on proof techniques, etc--, the additional space could be used to discuss the implications of the theoretical results in more accessible terms.
For example, a subject that ought to be discussed more clearly is the nature of the approximation induced by the lazy training regime. As far as I understand, this regime can be thought of as a sort of regularization that severely limits the capacity of the approximator.  Although the authors mention in the conclusion that “...convergence of lazy models may come at the expense of their expressivity”, after reading the paper I do not have a clear sense of how expressive such models actually are. In their experiments, Chizat et al. (2018) observed that the performance of commonly used neural networks degrades when trained in the lazy regime --to a point that they consider it unlikely that the successes of deep learning can be credited to this regime of training. It seems to me that this subject should be more explicitly discussed in a paper that sets out to provide theoretical support for deep reinforcement learning.
Still regarding the behavior of lazy approximators, my intuitive understanding is that they work as a linear model using random features. If this interpretation is correct, this makes the theoretical results a bit less surprising. They are still interesting, though, for they can be seen as relying on a “smoother” version of the linearity assumption often made in the related literature. Maybe this is also something worth discussing? Still on this subject, it seems to me that one potential disadvantage of lazy models with respect to their linear counterparts is that it is less clear how to enforce the lazy regime in practice. In Section 4.2 the authors discuss how this can naturally happen as a side-effect of the initialization, but it is unclear how applicable the particular strategy used to illustrate this phenomenon, with the “doubling trick”, is in practice. This is another example of a less technical discussion that would make the paper a stronger contribution.