We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model's outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.

The paper proposes an Energy-Based-Model (EBM) for scoring the possible configurations of amino acid side chain conformations in protein structures with known amino acid backbone structure. The energy of the side-chain conformation (the chi-angle) for a given amino acid in the structure is calculated as a function of a local neighbourhood of atoms (A), where each atom is embedded into a 256d vector using its cartesian coordinates, atom identity, atom side-chain position and amino acid identity. The model is trained using approximate likelihood where the model samples are generated using precalculated table (from literature) of possible Chi angles conformations conditioned on the back-bone amino acid identity and back-bone angles. The results seem comprehensive comparing the transformer based energy function parameterization with two sensible baselines as well as the Rosetta energy function  which is the de facto standard tool for these types of calculations. Using rotamer recovery accuracy as the benchmark measure the empirical results are close to performance as the Rosetta energy model however always slightly worse. Further visualizations of the energy levels for different Chi angles seems to support that the learned energy function captures well known characteristics of the rotamer configuration energy landscape.
Score:
Overall I think that the paper is solid and tackles an interesting problem of learning energy functions for physical systems from data. The experimental results are comprehensive and mostly supports the claims made in the paper. My main (slightly minor) concern about the paper is the obsession with discarding years of learned knowledge about handcrafted energy functions with fully learned functions. It seems to me that combining domain knowledge with a learned model should close the last small gap (and presumably surpass) the performance of e.g Rosetta. Combining I think the paper should be accepted at the ICLR.
Comment/Questions:
Motivation
Q1.1) Paper motivation: The paper several times seems to suggest that energy functions derived from physical knowledge are problematic (e.g. abstract). For many physical systems I don’t think this is true and would like the author's comments on why learned energy functions are preferable - arguably they can only capture properties present in the data and not any prior knowledge?
Q1.2) Given that the paper is motivated by fully learning an energy function from data without injecting any prior knowledge i would like the authors comment on the construction of the q(x|c) distribution. This is based on essentially a contingency table between Phi/Psi/Chi angles and to me seems like injecting prior knowledge about physically possible rotamer configurations into the learned procedure?
Method / Experimental Results:
Q2.1) With respect to the primary results in table 1) and table 2). The authors claim comparable results to the Rosetta Energy function (page 5. Sec 4.3, page 9, sec 6). However the experimental results are always slightly worse than the Rosetta energy function and I (strongly) suggest that the authors rephrase those statements to reflect that.
Q2.2 With Respect to Table 3) Firstly, Why are the only results for 16 amino acids in the table ? Secondly, just naively counting the Atom Transformer are better than Rosetta in 11 of 16 amino acids - this seems slightly at odds with the main result where the Atom Transformer is performing slightly worse than Rosetta?
Clarity
Q3): The notation in section 3 and 4 is slightly confusing. Especially I think q(x|c) is slightly misleading since it, (to my understanding) is a conditional distribution over Chi, conditioned on Psi/Phi/AminoAcid and not rotamer (x) and surrounding molecular context (c).


Abstract reasoning, particularly in the visual domain, is a complex human ability, but it remains a challenging problem for artificial neural learning systems. In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks. MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks. MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels. MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates. We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM). For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8%. For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of-the-art models by a considerable margin.

The paper proposes a novel, feedforward, end-to-end trainable, deep, neural network for abstract diagrammatic reasoning with significant improvements over the state of the art. The proposed model architecture is reasonable and is designed to exploit the information present at multiple granularities – at the level of objects in the diagram, their relations across diagrams, and diagram subsets. As a multimodule neural pipeline, it seems a reasonable design. Further, it shows significant performance gains over the state of the art.
However, the writing quality is poor and is the primary reason for my giving it a low score. The paper is difficult to read and it’s hard to figure out the terminology and it’s grounding in the problem; the high-level abstract design and design choices that address the nature of the problem from the low level details, etc.
The paper uses terminology without explaining the reason for it - for example, why is the approach called ‘Multiplex Graph Networks’? What information is being multiplexed and how? Graphs are conceptual in the proposed approach – there doesn’t seem to be any graph algorithms or graph based processing. Once the module is run for search space reduction, the set of edges or relations (node pairs) become well-defined (in adjacent rows, columns) as well diagram subsets (edge pairs). The corresponding modules are just computing vectorial embeddings. Similarly, there is no reasoning that’s taking place. Reasoning requires tokens and grammar over such tokens which is not there in this case.  The proposed model is non-interpretable.
The technical writing is loose and hand-wavy. The appendix is a lot of grammatical mistakes.
A few clarifications may be helpful:
- “The reasoning module can also be considered as another graph processing module”?
- “… we use spatial attention to iteratively attend …” – there is no iterative attention. It’s all parallel.
- What do the ‘N’ nodes in each layer correspond to? There are clearly not objects or diagram primitives as they can vary in number in each diagram.
- if interlayer connections are between objects in different layers (diagrams), what is this supposed to capture? Clearly, there may not be any unique correspondence between objects across diagrams.
- What’s a cross-multiplexing gating function? If it’s a known concept, please provide a reference else explain.
Finally, I’m open to revising my score upwards if it turns out that I’m the only one who had difficulty with the writing. The architecture design makes sense for the addressed class of problems (though the proposed network is non-interpretable and doesn’t do any reasoning nor uses graphs or graph based processing in a meaningful way), the results are good and the experimental evaluation sufficient.


Analyzing the behavior of neural networks is one of the most pressing challenges in deep learning. Binarized Neural Networks are an important class of networks that allow equivalent representation in Boolean logic and can be analyzed formally with logic-based reasoning tools like SAT solvers. Such tools can be used to answer existential and probabilistic queries about the network, perform explanation generation, etc. However, the main bottleneck for all methods is their ability to reason about large BNNs efficiently. In this work, we analyze architectural design choices of BNNs and discuss how they affect the performance of logic-based reasoners. We propose changes to the BNN architecture and the training procedure to get a simpler network for SAT solvers without sacrificing accuracy on the primary task. Our experimental results demonstrate that our approach scales to larger deep neural networks compared to existing work for existential and probabilistic queries, leading to significant speed ups on all tested datasets.

This paper approaches the problem of using logic-based reasoning tools to analyze binary neural networks. They propose general changes to BNN architectures at the neuron level so as to adapt them for more efficient usage by such tools. Their main contributions are a new design choice utilizing ternary, rather than binary, quantization, and additional variables to propagate through the network containing upper and lower bounds for each layer output. They also suggest further promising directions to adapt BNNs at the block and network level. Their experiments show that the proposed changes do not significantly alter test accuracy from a standard trained baseline model while dramatically reducing the number of model parameters. Consequently, they are able to demonstrate that SAT solvers can find adversarial examples much more quickly with their new architectural alterations.
Overall, this paper presents a novel contribution with an idea for more efficient BNNs and experimentally verifies the success of their proposed changes. However, there is a concern regarding the experimental setup.
A large part of the motivation for this task arises from the field of verification regarding adversarial examples. While binarized neural networks are not my area of expertise, I am familiar with a number of verification papers on full neural networks. As such, I find the datasets used, accuracies reported, and model size a bit peculiar. In ‘Scaling provable adversarial defenses’ (Wong ’18), the authors were able to verify models with a couple million parameters on both MNIST and CIFAR. In my understanding, the primary advantage of BNNs is their efficiency in having only binary weights. If this is the case, I see no reason why the experimental setup would have such a small model. Additionally, it is generally well-known that MNIST and its variants are not a particularly hard datasets to classify. Many different architectures can easily achieve 98 or 99 percent test accuracy. A quick literature search found ‘A review of Binarized Neural Networks’ (Simons ’19), which reports better numbers on MNIST and decent results on CIFAR-10 (and in fact, even ImageNet). Their experimental evidence crucially relies on the conclusion that their techniques allow for faster SAT solver computation *while* maintaining comparable test accuracy. As such, it is my opinion that to be a strong paper, they must show this property holds for larger networks and more significant datasets.
Additionally, a comment for the authors to consider: I would suggest switching some of the supplementary materials with the main content. The exposition on BNNs is quite dense and detailed. Some of these details could be moved to the appendix and replaced with more experimental graphs and images. This would provide some breaks in the text so as to provide ease of reading.


Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech. Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS -Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at http://tiny.cc/gantts .

I want thank the authors for solving this long-standing GAN challenge in raw waveform synthesis. With all due respect, previous GAN trials for audio synthesis are inspiring, but their audio qualities are far away from the state-of-the-art results. Although the speech fidelity of GAN-TTS is still worse than WaveNet and Parallel WaveNet from the posted sample, it has begun to close the significant performance gap that has existed between autoregressive models and GANs for raw audios. Overall, this is a very good paper with significant contributions to the filed.
Detailed comment:
1, In WaveNet, the conditional features (linguistic / mel-spectrogram) are added as bias terms in the convolutional layers. Did the authors tried this alternative architecture for the generator, which uses the white noisy z as network input (similar as flow-based models, e.g., Parallel WaveNet) and the conditional features as bias term in the convolutional layers?
2, Could the authors comment the importance of serval architecture choices in this work? From Table 1, it seems to me that the ensemble of random window discriminators is the most important (perhaps the only important) contributing factor for the success. For example, the MOS score was boosted from 1.889 to 4.213 by replacing a single full discriminator to the ensemble of RWDs.
3, The notations in Eq. (1) and (2) are messy. Although I can figure their meaning from the context, one may clarify certain notations if they appear at the first time.
4, The stable training (NO model collapses) is pretty impressive. Could the authors shed some light on the potential reason? Does the ensemble of RWD regularizes the training? What's your experience for training FullD (does not have random window ) and cRWD_1 (only has one random window discriminator)? Are they still very stable? Also, could the authors comment on the importance of large batch size -- 1024 for stable training of GAN-TTS?
5, Although there is a notable difference, one may properly mention previous work Yamamoto et al. (2019), which uses GAN as an auxiliary loss within ClariNet and obtains high-fidelity speech ( https://r9y9.github.io/demos/projects/interspeech2019/ ).
Yamamoto et al. Probability Density Distillation with Generative Adversarial Networks for High-Quality Parallel Waveform Generation. 2019.
=== update ===
Thank you for the detailed response.
2,  Thanks for the elaboration.
4,  It would be very interesting to see an analysis of model stability with smaller batch sizes.


Despite significant progress in Robotic Object Search (ROS) over the recent years with deep reinforcement learning based approaches, the sparsity issue in reward setting as well as the lack of interpretability of the previous ROS approaches leave much to be desired. We present a novel policy learning approach for ROS, based on a hierarchical and interpretable modeling with intrinsic/extrinsic reward setting, to tackle these two challenges. More specifically, we train the low-level policy by deliberating between an action that achieves an immediate sub-goal and the one that is better suited for achieving the final goal. We also introduce a new evaluation metric, namely the extrinsic reward, as a harmonic measure of the object search success rate and the average steps taken. Experiments conducted with multiple settings on the House3D environment validate and show that the intelligent agent, trained with our model, can achieve a better object search performance (higher success rate with lower average steps, measured by SPL: Success weighted by inverse Path Length). In addition, we conduct studies w.r.t. the parameter that controls the weighted overall reward from intrinsic and extrinsic components. The results suggest it is critical to devise a proper trade-off strategy to perform the object search well.

This paper proposed a hierarchical approach to perform robotic object search (ROS).
The idea is to use a high-level policy which produces subgoals and a low-level policy which produces atomic actions conditioned on both the subgoals and the true goal, and which is trained with a weighted sum of the original extrinsic reward and a reward for reaching the subgoal. Subgoals are consist of different objects in the field of view which the robot can choose to approach.
The approach is evaluated on the House3D dataset, where it is shown to perform well.
Recommendation: weak reject.
This isn't a bad paper, but I'm not sure it will be of broad interest at ICLR.
It is very specific to ROS problem and House3D dataset and doesn't seem to propose a general algorithm which can be broadly applicable elsewhere. The mechanism for generating subgoals and training the low-level policy is very task dependent (subgoals are constrained to be objects in the field of view, the intrinsic reward for training the low level policy is dependent on the size of the bounding box of the object defining the subgoal). While this probably accounts for improved performance on the House3D dataset, I think the audience of ICLR would be more interested in a general approach which can be used in many different domains (even at the cost of performing less well on a specific domain than something more tailored). This paper may be a better fit for a robotics conference.
Another point concerning the experimental evaluation. Sparsity of the rewards is mentioned as a main motivation for the hierarchical approach. However, there are a number of methods which use exploration bonuses to address this issue (pseudocounts, random network distillation, ICM etc. [1, 2, 3]). At least one of these should be included as a baseline.
Specific comments:
- using two letters for denote a single variable is confusing, since it seems like a product. I.e. using "sg" to denote a subgoal, "at_t" to denote area. Please use a single letter and add subscripts if necessary to disambiguate.
- in the various equations, please use "\log" instead of "log" so that it is not italicized.
- bottom of page 4: "Q-leaning" -> "Q-learning"
- page 2: "way pf" -> "way of"
- please use more informative names for Settings A/B
- First paragraph in Section 2: "hierarchical policy for the robot to perform the object search, motivated by how human beings conduct object search". Saying the method is similar to how humans behave is a fairly big claim that should be substantiated by appropriate references, or not made at all.
References:
[1] https://arxiv.org/abs/1810.12894
[2] https://arxiv.org/abs/1703.01310
[3] https://arxiv.org/abs/1705.05363


Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years. However, such complexity can be misleading and hides over-fitting. We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization. We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images. Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.

This paper presents an empirical study of using error reduction as a curiosity measure. The authors consider an auto-encoder model, a colorization model and RND as intrinsic motivation signals. I find the write up very unclear and have trouble understanding what the claims are and how they are backed up.
Major points:
* About the claims as stated on page 2: 1) The first claim I don't understand, I think what is meant is on navigation tasks they find "their measure of representation learning" (proposed in kolesnikov 2019) seems to correlate well with reward optimization. In 3.1 to back this claim they claim to test disentanglement but seem to test classification. I see no reason to not put that part in 4.1. 2) The authors claim to propose a new method: it can't be a separate representation network to derive rewards because that is what burda et al. and many many others do, it can't be the minimax formulation because that has been known for a while (e.g. predictability minimization schmidhuber) so I am not sure what the claim is about. What exactly is novel about the model. 3) CRL seems at best to outperform baselines on beamrider, qbert and riverraid but the results are impossible to assess. We don't know what the x axis is in figure 7 (is it frames, with or without repeats, is updates etc). Pong -12 is far from learnt for instance and its one of the easiest games.
* suprisal objective and modeling objectives are very high level concepts that we can talk about in the introduction and conclusion but much more precise terms need to be employed in the model exposition. The readers need to know what sort of properties they should have ideally easily identify examples (without having to read 2 papers and a large survey). I would start with the minimax formula and then explain what is considered, how the different intrinsic rewards are added, if they are normalized, how they are weighted etc. etc.
* the details about failed experiments, environments and architecture should IMO be relegated to experiments
* It should be very clear early on that the model is separate from the representation.
* auto-encoders are a large family of models and it is not clear from the paper which exact model is meant by the authors. Also, citing Bengio 2013 is NOT a valid citation for auto-encoders. The right citation depends on the model you apply please use that!


Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms, such as data noising, RAML, and softmax policy gradient, have also been developed from different perspectives. In this paper, we present a formalism of entropy regularized policy optimization, and show that the apparently distinct algorithms, including MLE, can be reformulated as special instances of the formulation. The difference between them is characterized by the reward function and two weight hyperparameters. The unifying interpretation enables us to systematically compare the algorithms side-by-side, and gain new insights into the trade-offs of the algorithm design. The new perspective also leads to an improved approach that dynamically interpolates among the family of algorithms, and learns the model in a scheduled way. Experiments on machine translation, text summarization, and game imitation learning demonstrate superiority of the proposed approach.

This paper claims to propose a general entropy regularized policy optimization paradigm. MLE and RL are special cases of this training paradigm. Paper is well written, and the experimental results are convincing enough.
However, there are still some minor problems in the paper. For the optimization framework ERPO (shown in Equation 1), it consists of three parts, a cross-entropy term (Shannon entropy), a $p,q$ KL divergence term, and a reinforcement learning reward loss item. From the framework point of view, it is not like the author claim that is supposed to present a general optimization framework, including various optimization algorithms. Instead, it is just a combined loss through weight control and the selection of corresponding functions. It may not really theoretically work to unify various types of optimization algorithms for general cases, let alone claiming that this is a general optimization algorithm framework.
For the interpolation algorithm (I regard this is the true technical contribution of this paper), the authors used an annealing mechanism to use different weights and functions at different stages of training. The essence is that after MLE pre-training, different optimization algorithms are used in different stages, and this should be the focus of the article. The annealing settings used is only introduced in the appendix simply. Without more comparison experiments, we cannot clearly get the conditions for the annealing algorithm to be effective and ineffective.
For the title of connecting the dots between MLE and RL, this paper did not do so, MLE and RL are only used collaboratively, and this has also been mentioned in previous work.
typo
Page 6 Paragraph “Other Algorithms & Discussions”: We We show in the appendix… -> We show in the appendix…


Data have often to be moved between servers and clients during the inference phase. This is the case, for instance, when large amounts of data are stored on a public storage server without the possibility for the users to directly execute code and, hence, apply machine learning models. Depending on the available bandwidth, this data transfer can become a major bottleneck. We propose a simple yet effective framework that allows to select certain parts of the input data needed for the subsequent application of a given neural network. Both the associated selection masks as well as the neural network are trained simultaneously such that a good model performance is achieved while, at the same time, only a minimal amount of data is selected. During the inference phase, only the parts selected by the masks have to be transferred between the server and the client. Our experiments indicate that it is often possible to significantly reduce the amount of data needed to be transferred without affecting the model performance much.

The paper addresses the problem of high-cost transfer between server and user for machine learning applications. The method proposes to augment inputs with channel/spatial masks that are trained via the Gumbel-softmax trick together with the model's weights trained/finetuned to account for the loss of information. These learned masks are then applied to the image before sending it to a server with where inference takes place to reduce file transfer costs. In the experimental study, the paper shows that on computer vision tasks inputs can be reduced with relatively little drop in accuracy and analyses how hyperparameters of the model affect its performance. The framework is also adapted to the downstream task-guided choice of compression techniques, e.g. which compression quality to choose for JPEG that will be an optimal trade-off in terms of downstream task quality vs data transfer cost.
Overall, this paper could be a valid algorithmic contribution, however, I have concerns about how this method fares with others, resolving the following issues in the author response will likely increase the score.
- A discussion on the connection/comparison with representation learning and dimensionality reduction (VAEs, quantization, etc) would help improve the exposition of the paper and help define how and when the suggested method is more appropriate to use, as well as citations to the other lines of work in reducing the model size (knowledge distillation [1], tensor decomposition approaches [2], adaptive computation time techniques [3], etc).
- I would like to see an experiment that compares other works mentioned as related work. [4] has been mentioned as one of the nontrivial compression methods for image data, how does the proposed method compare to it?
- Another experiment comparing a method from representation learning, e.g. a VAE trained with the embedding size corresponding to some optimal Q value in this work would be helpful.
- Please include reference accuracy values for the dataset/NN pairs used in Table 1.
- In Section 4.5, the paper states that large lambda values correspond to blue and red lines, however in the corresponding figure large lambda values correspond to blue and orange lines which exhibit different behaviors, could you please clarify this?
References
[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network."
[2] Novikov, Alexander, et al. "Tensorizing neural networks." Advances in neural information processing systems. 2015.
[3] Figurnov, Michael, et al. "Spatially adaptive computation time for residual networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
[4] Jiang, Feng, et al. "An end-to-end compression framework based on convolutional neural networks." IEEE Transactions on Circuits and Systems for Video Technology 28.10 (2017): 3007-3018.


Deep neural networks (DNNs) have achieved unprecedented practical success in many applications. However, how to interpret DNNs is still an open problem. In particular, what do hidden layers behave is not clearly understood. In this paper, relying on a teacher-student paradigm, we seek to understand the layer behaviors of DNNs by "monitoring" both across-layer and single-layer distribution evolution. Here, the "across-layer" and "single-layer" considers the layer behavior along the depth and a specific layer along training epochs, respectively. Relying on optimal transport theory, we employ the Wasserstein distance (W-distance) to measure the divergence between the layer distribution and the target distribution. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth. ii) For a specific layer, the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations. iii) However, a deep layer is not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs. Extensive experiments justify our theoretical findings.

This paper presents a method to compute the distance of distribution of two layers in neural networks by using the label distribution mapping (e.g., Frogner et al., 2015). With the tool, authors could see how individual layers could related each other across-layer (along the depth) and single layer (training epoch).
I believe that the contributions of this paper are week in analyzing individual layers across-layer since there are many extensive studies are conducted on information bottleneck methods with mutual information. I believe that those methods are better to analyze the dynamics of learning even without the additional label distribution mapping.
However, authors of this paper presents a way to utilize the label distribution mapping to compare the distance of individual layers when an input image come as shown in Figure 5 which I believe the main contribution of this paper.
The (somehow artificial and ambiguous) term, label distribution is used several places before it is defined. Even in Section 3, the label distribution mapping is not clearly explained except for the description of FC+softmax. Thus, it would be better to clarify the definition. Also, it is not clear that the label distribution reflect the actual distribution of (nodes or feature maps) in a specific layer. It would be good to spend more space and resources (e.g., image and/or running examples) to explain the definition of label distribution.


We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment. Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process. We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal. Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network). Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer, and Otto (1998).

This paper proposes that we learn the “arrow of time” for an MDP: that is, a function (called the h-potential) that tends to increase as the MDP steps forward. Such an arrow should automatically capture notions such as irreversibility, and so can be used to define a measure of reachability, which previous work has shown can be used to penalize the agent for causing negative side effects. In addition, it can be used as intrinsic motivation for the agent: in particular, the agent can be rewarded for trajectories that decrease the h-potential (i.e. are “like” going backwards in time, or reducing entropy), which is hard to do and should lead to interesting skills. They propose that we learn the arrow of time by optimizing a function to grow over time along trajectories take from a random policy. Experiments demonstrate that in simple environments the learned function has the properties we would expect it to given results from physics.
I am conflicted on this paper. I like the novelty of the suggestion; it is not something I would have expected to see in an ML paper. The discussion and experiments have convinced me that the idea is worth investigating: they show that the learned arrow of time approximately satisfies the properties we would expect, and demonstrate their two use cases in two simple environments: the intrinsic reward is used in a tomato-watering environment, while the side effect avoidance is shown in Sokoban (though the experiment only shows that the h-potential increases with irreversible actions -- it doesn’t actually use the h-potential to create an agent that reliably avoids irreversibly pushing boxes into corners). However, it’s not clear to me whether or not the method would scale to more complex environments, the current experiments are more like demonstrations (there are no baselines), and the paper is hard to understand without a background in physics. Overall, given the novelty of the suggestion, I lean towards a weak accept.
----
In your objective, you use an expectation over the timestep in the trajectory. Why not instead take an average over all the timesteps in the trajectory? This should be equivalent. (If trajectories are all the same length, you could also take a sum over the timesteps.) Similarly, in the algorithm, why do you sample from the dataset? It would likely be better to randomly shuffle the dataset (at the timestep level) once, and then iterate through the dataset computing gradient updates. (This is standard practice in supervised learning.)
----
When scaling the algorithm up to larger environments, you will likely run into the problem that uniform policies are often very bad at exploring the state space. Consider for example the Overcooked environment (https://bair.berkeley.edu/blog/2019/10/21/coordination/ ). The environment is not dissipative, so you’d expect the h-potential to be zero. However, uniform policies are extremely unlikely to ever deliver a soup (which requires several hierarchical actions), but sometimes will pick up an onion. If you don’t know about soup delivery, then picking up an onion looks irreversible, because there’s no way to get rid of it, and the h-potential will rise. I think it would significantly improve this paper to demonstrate a solution to this problem.
One possibility is to redefine the arrow of time to be with respect to some distribution over states: E_{s ~ p(s)} E_{a ~ Uniform(A)} E_{s’ ~ p(s’ | s, a)} [ h(s’) - h(s) | s ]. Initially, your distribution over states can be the initial state distribution. But then you can use the learned arrow of time as an intrinsic reward to find a policy that finds “interesting states”. In Overcooked, we would hope that this policy learns to pick up onions. Then, your new distribution over states can be the states reached in 0-20 timesteps when following the new policy, that is, you collect your dataset by running the “interesting” policy for some time, and then switch back to the uniformly random policy, and only use the states / actions collected during the uniform random policy as part of your dataset. Hopefully, this would include states where the onion is placed in a pot, and the h-potential would learn that placing an onion in a pot is “irreversible”. Then, another round of using the h-potential would lead to a policy that places onions in pots. Collecting data could then discover delivering soups, and so on.
----
In addition to experiments on larger environments, I would like to see better experiments for the two intended use cases. For example, can you use the learned arrow of time to solve the environments in (Krakovna et al), and how does it compare to relative reachability and its many variants? Similarly, how does your intrinsic reward compare to existing exploration methods (of which there are many, but consider count-based methods, curiosity (Pathak et al), random network distillation (Burda et al))?
----
(This section did not affect my assessment of the paper)

Have you considered finding theoretically what it means to take the difference between the h-potential of two states (i.e. your reachability measure)? I could imagine that the answer is something like “reachability(s, s’) is proportional to the log probability of reaching s’ from s when acting according to a uniformly random policy”. Perhaps this has to be normalized against the log probability of reaching other states from s. This would be very interesting as a potential definition of reachability.
----
There is a lot of jargon from physics that will not be familiar to the typical audience at ICLR (e.g. Hamiltonian, Liouville’s theorem, Maxwell’s demon, free energy functionals, etc.) I would recommend improving the clarity of the paper on this axis. There’s no particular need to name Maxwell’s demon prominently -- the exposition is sufficient by itself, perhaps Maxwell’s demon can be mentioned in a footnote. Consider adding a discussion of ergodicity (as applied to Markov chains / MDPs), which will be more familiar to the audience and serves a similar purpose as the discussion on Hamiltonian systems. Perhaps move the experiment with the free energy functional to the appendix, and move the experimental details for the other experiments from the appendix to the main paper.


Generic responses are a known issue for open-domain dialog generation. Most current approaches model this one-to-many task as a one-to-one task, hence being unable to integrate information from multiple semantically similar valid responses of a prompt. We propose a novel dialog generation model that learns a semantic latent space, on which representations of semantically related sentences are close to each other. This latent space is learned by maximizing correlation between the features extracted from prompt and responses. Learning the pair relationship between the prompts and responses as a regression task on the latent space, instead of classification on the vocabulary using MLE loss, enables our model to view semantically related responses collectively. An additional autoencoder is trained, for recovering the full sentence from the latent space. Experimental results show that our proposed model eliminates the generic response problem, while achieving comparable or better coherence compared to baselines.

1. Summary: The authors proposed to alleviate the generic response problem in open-domain dialog generation by generating a response to a prompt from a semantic latent vector. This vector needs to be located close to the latent vector of the corresponding prompt. To this end, they employ canonical correlation analysis and auto-encoder to learn the mapping from a sequence of text to a semantic latent vector. To model the variations and topic shifts that may happen in responses, they also use a separate intermediate vector in the auto-encoder of generating responses.
2. Overall assessment: While this paper is quite fun to read, it is not innovative enough and it lacks some critical experiments and error analysis to be accepted this time. I'll elaborate on these problems in my comments below.
3. Strengths:
3.1 This paper is well written. The motivation and the main idea are well explained and pretty easy to understand. Although there are some details missing, it doesn't prevent readers from understanding and enjoying this paper.
3.2 The use of human evaluation is a great plus. Subtle characteristics of text, such as readability, coherence, and specificity cannot be well justified by automatic evaluation. Human evaluation is a must for these aspects. It's great to see the authors include this in this paper.
4. Weakness and questions:
4.1 It seems the model in this paper can be connected to adversarial learning from some angles. It would be great to see such analysis in this paper.
4.2 The experiments in this paper do not look thorough enough. There are many more things should be included, such as error analysis, comparisons over different variations of the proposed model and so on. What if we remove $Y_u$ in both training and inference but keep the overall model the same? How important is $Y_u$ and what's its effect? Many such questions haven't been answered in this paper.
4.3 Embedding average cosine similarity seems to be too simple to be used for evaluation as it omits the contextual information and semantic meaning of a sentence.
4.4 It would be interesting to see the Dist-1 and Dist-2 scores of the gold standard responses. It's a good reference to let us know ho diversified the real responses are.
4.5 I'd like to see some explanation of how the raters rate specificity and coherence. What are the standards they use? What are the instructions the author give to them?
4.6 Are improvements significant in Table 1? It seems the proposed model is not performing very stably over different metrics. Some anlysis on this would be great.


We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant filters from an over-parameterized network. Our algorithm uses a small batch of input data points to assign a saliency score to each filter and constructs an importance sampling distribution where filters that highly affect the output are sampled with correspondingly high probability. In contrast to existing filter pruning approaches, our method is simultaneously data-informed, exhibits provable guarantees on the size and performance of the pruned network, and is widely applicable to varying network architectures and data sets. Our analytical bounds bridge the notions of compressibility and importance of network structures, which gives rise to a fully-automated procedure for identifying and preserving filters in layers that are essential to the network's performance. Our experimental evaluations on popular architectures and data sets show that our algorithm consistently generates sparser and more efficient models than those constructed by existing filter pruning approaches.

Summary:

In this paper, the author propose a provable pruning method, and also provide a bound for the final pruning error.  Among most heuristics prune method,  pruning with mathematics guarantee is indeed more convincing. We expect this work can help people devoting some effort into more solid theoretical study in understanding the over-parameterized training.
Intuitively speaking, the sensitive neuron has greater contribution for the final output, reusing the corresponding filter and carefully rescale its value require many empirically attempts. To achieve a more reasonable algorithm, author prune the redundant channel by controlling the deviation of the summation statistically small,  and reusing the filter by important sampling the given channel. Experiment show that this method can reach a competitive prune radio against other pruning algorithm, and show robustly in retained parameters vs error experiment.
Weakness:
1. experiment is too weak
ImageNet model has great impact on most CV problem, and the current release models are flooding in the open source world. Author should at least provide a imagenet model and make this work more convincing. Besides, Author should also consider an experiment in modern lightweight network, vgg and resnet like model are out of fashion and so big that any one can make a sound result on it.
2. lack of a comparing experiment for random select the top-k norm.
Important sampling require an input of probability [p1, p2, p3, ... pn],  if those probabilities are nearly uniform, important sampling will behave like a random sampling method. In most case, if we want to prune the large channel network,  picking the top-1 significant filter or random sampling top-k filter will almost do the same thing.
3. lack of further theory consideration
author only consider the single layer reconstruction, without discussing the overall accumulative error. Unlike the other deterministic method, sampling skill suffer variance propagation problem, the pre-layer variance will affect the sampling probability of next layer, how this pruning work if we change status of the pre-layer,  I didn't find any theoretical guarantee and only find a proof of single layer reconstruction bound.


Understanding the inductive bias of neural networks is critical to explaining their ability to generalise. Here, for one of the simplest neural networks -a single-layer perceptron with n input neurons, one output neuron, and no threshold bias termwe prove that upon random initialisation of weights, the a priori probability P (t) that it represents a Boolean function that classifies t points in {0, 1}
n as 1 has a remarkably simple form: P (t) = 2 −n for 0 ≤ t < 2 n . Since a perceptron can express far fewer Boolean functions with small or large values of t (low "entropy") than with intermediate values of t (high "entropy") there is, on average, a strong intrinsic a-priori bias towards individual functions with low entropy. Furthermore, within a class of functions with fixed t, we often observe a further intrinsic bias towards functions of lower complexity. Finally, we prove that, regardless of the distribution of inputs, the bias towards low entropy becomes monotonically stronger upon adding ReLU layers, and empirically show that increasing the variance of the bias term has a similar effect.

The topic of the paper is the inductive bias of neural networks. The authors study a simple model, namely a perceptron with no bias term viewed as a mapping from {0,1}^n->{0,1}. They show that initializing weights with a distribution that is symmetric under coordinate sign flips corresponds to an initialization in function space that is biased towards low-entropy functions. They also exhibit empirical evidence that by adding a bias term, or by using multiple layers, this tendency towards low entropy appears to increase. They also prove a bound on the minimal size of a network in order for it to represent all boolean functions. Finally, they prove a result that suggests that for ReLU networks with infinite widths the bias towards low-entropy function does indeed increase with depth.
My main concern regarding this paper is that the claim in the title and the statement of Theorem 4.1 seem to rely crucially on the fact that the functions are viewed with input as {0,1}^n. The origin of the "simplicity", in the basic case that the authors address (perceptrons with no bias) appears to be a consequence that hyperplanes through the origin are quite likely to classify input points in {0,1}^n similarly. If one switches to a symmetric domain, for example {-1,1}, the effect in this setting completely disappears. The authors actually mention this in Section 5, noting that the expressivity of the perceptron is much lower for centered inputs. However, this to me suggests that Theorem 4.1 is not capturing any significant aspect of neural networks (in fact, the statement is a property of how linear hyperplanes to separate {0,1}^n, not neural networks). I may be mistaken, but I would like the authors to clarify this point.
Another concern is related to Theorem 5.5. This might be a more substantial result, but it is difficult to interpret and its implications are not discussed. Understanding the effect of depth on the "simplicity bias" seems to me an important problem, but for some reason Theorem 5.5 (which deals with deep neural networks rather than linear perceptrons) is emphasized much less than Theorem 4.1. Why is this the case?
The paper is well-written but not always very clear. In particular, notation is not always defined and the authors use on notions from complexity theory that are not introduced (e.g., Lepel-Ziv complexity).
Other comments:
* Is the set F_t is defined as set of all functions with assigned \mathcal T, but later this seems to be restricted to the functions expressible by a network/perceptron.
* Definition 3.5: this defined the entropy H(f) of a function but then write H(p). It should probably be H(f) = -plog p - (1-p)log(1-p) where p = \mathcal T(f), right?
* Definition 3.6: some context or references for this definition could be useful.
* Regarding the fact that functions in F_t are not uniform, shouldn't the distribution be the same for isotropic weight distributions? Assuming the distribution of w/|w| is uniform on the sphere, a more precise description of  P(f) seems possible
* Section 4.3: what is the "rank" in this setting? Isn't the parameter a vector w in R^n?
* Some typos: Definition 3.1 w_l \in R^{n_{l+1}}, ",." in the beginning of Section 5, several in the last paragraph of Section 5.


Statistical generative models for molecular graphs attract attention from many researchers from the fields of bio-and chemo-informatics. Among these models, invertible flow-based approaches are not fully explored yet. In this paper, we propose a powerful invertible flow for molecular graphs, called graph residual flow (GRF). The GRF is based on residual flows, which are known for more flexible and complex non-linear mappings than traditional coupling flows. We theoretically derive non-trivial conditions such that GRF is invertible, and present a way of keeping the entire flows invertible throughout the training and sampling. Experimental results show that a generative model based on the proposed GRF achieve comparable generation performance, with much smaller number of trainable parameters compared to the existing flow-based model.

Summary:
The paper proposes a flow-based model for generating molecules, called Graph Residual Flow (GRF). GRF is a version of iResNet that is tailored to molecule generation. The experiments show that GRF achieves similar performance to GraphNVP (a previous flow-based model for molecule generation), but has about 100 times fewer parameters.
Decision:
This is interesting work, but in my opinion the current version of the paper needs significant improvement. For this reason, my recommendation is weak reject.
Contribution & originality:
I think the application of flow models to the generation of molecules is an interesting research direction. Since molecules are structured objects that can be represented as graphs of connected atoms, care has to be taken to design suitable generative models for them, and this paper is a step in this direction.
The proposed model (GRF) is similar to iResNet in most aspects. The main differences are:
- The linear layers in iResNet are modified to take into account the connectivity of the molecule.
- Separate networks are used to generate the adjacency tensor and the feature matrix. The adjacency tensor is generated first, and then the feature-matrix network is conditioned on the generated adjacency matrix.
Other than the above, GRF is a straightforward application of iResNet in molecule generation.
Technical & writing quality:
The experiments are interesting, and GRF is shown to perform similarly to GraphNVP while using 100 times fewer parameters. The comparison between GRF and GraphNVP is in my opinion the most interesting part of the paper. The experiments that show GRF can reconstruct molecules exactly and that generated molecules are a smooth function of the latent variables are less interesting in my opinion; exact invertibility and smoothness are inherent properties of flow-based models that exist by design, so there isn't anything surprising about them.
The main issue with the paper is that it's poorly written. In my opinion, the writing quality of the paper needs significant improvement for the paper to qualify for publication. The model is explained very poorly; if I weren't already familiar with iResNet I don't think I would be able to understand the model and be able to reproduce the results by reading the paper. Moreover, the use of English throughout the paper is poor: there are a lot of grammatical mistakes (such as incorrect usage of articles or lack of subject-verb agreement), and awkward expressions. In the following I will do my best to make concrete suggestions for improvement, but I would strongly encourage the authors to revise the paper thoroughly and have it checked for grammar and writing quality.
Suggestions for improvement:
"The decoders of these graph generation models generates" --> generate
"The state-of-the-art VAE-based models [...] have good generation performance but their decoding scheme is highly complicated"
In what way is it complicated? Please provide more details.
"invertible flow-based statistical models [...] does" --> do
"[flow-based models don't] require training for their decoders because the decoders are simply the inverse mapping of the
encoders"
This is an odd and unusual description of flow-based models. The way I see it, flow-based models aren't autoencoders, but functions that reparameterize data (here molecules) in terms of simpler random variables (here Gaussian noise). Saying that flow-based models don't require training for their decoders makes little sense.
"Liu et al. (2019) proposes" -->  propose
"GraphNVP has a serious drawback when applied for sparse graphs" --> applied to sparse graphs
"only one affine mapping is executed for each attribute of the latent representation"
There are also coupling layers that are non-affine and therefore more flexible. For example:
- Neural Autoregressive Flows, https://arxiv.org/abs/1804.00779
- Sum-Of-Squares Polynomial Flow, https://arxiv.org/abs/1905.02325
- Neural Spline Flows, https://arxiv.org/abs/1906.04032
"show that a generative model based on the GRF has much fewer trainable parameters compared to the GraphNVP, while still maintaining a comparable generation performance"
If GRF can achieve a performance similar to GraphNVP with fewer parameters, shouldn't it be able to achieve much better performance with more parameters? If this is true, I would have expected to see an experiment were GRF uses more parameters and as a result outperforms GraphNVP. If GRF doesn't outperform GraphNVP even with more parameters, then I would expect to see a discussion explaining why.
it would be good to explain fully the format of the data as early as in section 2.1. For example, it wasn't clear to me until much later how a flow-based model can be applied to discrete data, and whether the rows of the adjacency tensor and feature matrix are one-hot or not. Currently, the format of the data is partially explained three times (in section 2.1, section 3.1 and section 4.1) and only in section 4.1 things become clear for the reader.
Beginning of page 3: the dequantized matrix X' is used before having been defined. This is an example of why the format of the data should be fully explained early on.
"The abovementioned formulations map only those variables [..] (Eqs.(2,5), and the remaining nodes [..] (Eqs.(3,5)"
The first reference should be Eqs. (2,4). Also, the parentheses should close.
"ResNets [...], the golden standard for image recognition"
The expression is "gold standard" not "golden standard". Also, it's too strong a statement and rather subjective to say that ResNets are the gold standard for image recognition; better say that ResNets have been successful in image recognition, which is an accurate and objective statement.
Section 2.2 introduces iResNets, but it doesn't explain clearly that making the residual block contractive is a sufficient condition for invertibility. This is a crucial element for understanding iResNet that is used later on in the paper, so it should be clearly explained early on.
"i-ResNet [...] presents a constraint regarding the Lipschitz constant"
Too vague, and it doesn't explain what the constraint is.
"MintNet [...] derives the non-singularity requirements of the Jacobian of the (limited) residual block"
Also vague and not informative.
Beginning of section 3: GraphMVP --> GraphNVP
"we cannot directly apply the change-of-variables formula directly"
Uses "directly" twice.
"we do not take it consideration" --> we do not take into consideration
"an tensor entry X'[i, m] is a neighborhood" --> a tensor entry X'[i, m] is a neighbor
"Similar discussion holds for A'"
It's unclear to me what exactly holds for A', please be more specific. The previous statement mentioned neighbours, how does this apply to A'?
In the text after eq. (10), R_X uses a different font.
"is subject to update in every layers" --> in every layer
Section 3.3. is particularly poorly written. Given that this is one of the main contributions of the paper, it is really important that this section is written well and clearly. Currently, the section contains two one-sentence paragraphs that are clearly out of place. Also, eq. (12) makes little sense. In particular:
- Isn't A a tensor of shape NxNxR? How did it become a matrix?
- In what sense is X a matrix representation of z_x? How is X computed? Shouldn't X be of shape NxM? Also, please use a different symbol other than X, as X is already used for the feature matrix.
- What is D and how is it defined?
- Is vec() the vectorization operator? Does that mean that the input and output to the residual block has been flattened? This has not been explained or mentioned anywhere.
- Is W a fully-connected linear layer or a convolution as in iResNet?
- Where did the term D^{-0.5} A D^{-0.5} come from? How is it motivated / derived? Why does it make sense to use it? I understand that it's related to the graph Laplacian, but more information is needed for readers who are not familiar with graphs.
"is a Lipschitz-constant of for a certain function" --> of a certain function
The explanation of eq. (13) is very poor, and I don't think a reader who is not already familiar with iResNet can follow it. Also, how was the infinite sum approximated? Did you use truncation or the Russian-roulette estimator?
Section 3.4 introduces the term "atomic feature tensors" which is not defined. I presume it's the same as the feature matrix. Please be consistent with terminology throughout the paper.
"adjacent tensor" --> adjacency tensor
"we have configure" --> configured
Section 3.4.2 finally explains the invertibility condition. Clearly, this is an important element of the algorithm, and should be explained fully and clearly early on. By this point, the invertibility condition should be clear, and no further explanation should be needed.
"we selecte" --> selected
"satisfies the differentiability" --> "satisfies the differentiability condition", or even better "is differentiable"
What does "kekulized" mean?
"conmprises" --> comprises
"we adopt node-wise weight sharing for QM9 and low-rank approximation and multi-scale architecture"
These are important aspects of the architecture that are nowhere else mentioned, and should be explained more thoroughly.
"p_z(T_X, T_A; z)"
Shouldn't this be p_z(z; T_X, T_A)?
What does "unique molecules" mean? Are they the molecules that don't exist in the data? How do they differ from "novel molecules"?
It would be good to provide additional results. In particular:
- It would be good to report training/validation/test log likelihood, or learning curves, that compare between GRF and GraphNVP.
- It would be good to show how the metrics V, N, U, R in tables 1 and 2 vary with respect to the temperatures (for example, show curves of the four metrics as temperatures vary). Currently, results are reported for only one setting of temperatures, which doesn't show how sensitive the performance is to the temperatures.
The temperatures for ZINC-250k are really low (0.15 and 0.17), which to me is an indication that the flow model may have fitted the data quite badly.
How does GRF compare to GraphNVP in terms of generation time? How slow is it to generate from GRF given that it takes 30 iterations for each residual block?
What is GraphLinear? Maybe give a citation?
"Since one of the most GPUs currently used"
This phrase doesn't make much sense.
"principle components analysis" --> principal components analysis
"Principle" has a different meaning from "principal".
Figure 3: Wouldn't it be more informative to centre the visualizations at the mean and use the two principal axes, rather than random mean and axes?


Deep neural networks were shown to be vulnerable to crafted adversarial perturbations, and thus bring serious safety problems. To solve this problem, we proposed AE-GAN +rs , a framework for purifying input images by searching the closest natural reconstruction with little computation. We first build a reconstruction network AE-GAN, which adapted auto-encoder by introducing adversarial loss to the objective function. In this way, we can enhance the generative ability of decoder and preserve the abstraction ability of encoder to form a self-organized latent space. In the inference time, when given an input, we will start a search process in the latent space which aims to find the closest reconstruction to the given image on the distribution of natural data. The encoder can provide a good start point for the searching process, which saves much computation cost. Experiments show that our method is robust against various attacks and can reach comparable even better performance to similar methods with much fewer computations.

Summary: This paper proposes AE-GAN+sr, an auto-encoder based GAN for equipping neural networks with better defenses against adversarial attacks. The authors evaluate their method on black-box attacks, white-box attacks, and gray-box attacks on MNIST and Fashion-MNIST, and show decent empirical results when compared to baselines.
Decision: Reject. The writing was hard to follow and the experimental evaluations could have been stronger.
Supporting Arguments/Feedback:
- There are a lot of phrases which should be reworded in the text (e.g. “it requires attention to” in paragraph 1, “detect if sample is from a normal distribution” at the end of Section 1, etc.) for better clarity. The writing was pretty hard to follow and the paper would really benefit overall after it has been improved.
- It was hard for me to tell from the text how exactly the models were implemented and trained (see Question #1). Making this point more clear, in either the main text or the Appendix, would be really helpful.
- The performance of AE-GAN+sr as evaluated on MNIST and FashionMNIST did not provide convincing evidence that it outperformed baselines such as Defense-GAN or adversarial training, though I did appreciate the authors’ additional analysis on the tradeoffs between computational cost and performance for the Defense-GAN. The results would also have been more compelling had the authors evaluated their method on more complex datasets such as CIFAR-10.
Questions:
- It wasn’t clear to me how the encoder-assisted search process (Section 3.4) works in practice. When you use gradient descent to find the best encoding for a corrupted input, how many steps do you need to take? Does this step happen while the autoencoder and GAN are being trained, or are those pre-trained and you just take additional gradient steps in the latent space?
- Given that the BiGAN also incorporates an encoder (and they were also benchmarked against in the experiments), where do the advantages of the AE-GAN+sr come from? I think making this point in the text would also be helpful as well.


Adversarial examples are crafted with imperceptible perturbations with the intent to fool neural networks. Against such attacks, adversarial training and its variants stand as the strongest defense to date. Previous studies have pointed out that robust models that have undergone adversarial training tend to produce more salient and interpretable Jacobian matrices than their non-robust counterparts. A natural question is whether a model trained with an objective to produce salient Jacobian can result in better robustness. This paper answers this question with affirmative empirical results. We propose Jacobian Adversarially Regularized Networks (JARN) as a method to optimize the saliency of a classifier's Jacobian by adversarially regularizing the model's Jacobian to resemble natural training images. Image classifiers trained with JARN show improved robust accuracy compared to standard models on the MNIST, SVHN and CIFAR-10 datasets, uncovering a new angle to boost robustness without using adversarial training examples.

This paper proposes a novel regularization strategy for improving the robustness of networks to adversarial noise. A term is added to the standard supervised cross-entropy loss that encourages the Jacobian of the network to itself be interpreted as a valid image. This "regularization" term is constructed by running the input-output Jacobian of the classification network through an "adapter network" and then in turn interpreting its output as a "generator" in a GAN setup. A separate discriminator network is training to distinguish real input images from these adapter-processed input-output Jacobians. The overall regularization is the standard minimax GAN loss applied to this generator/discriminator setup. The impetus for this stems from a previous observation that salient or interpretable input-output Jacobians naturally arise for networks that have undergone adversarial training to increase robustness.
Although this whole setup seems to be a little "Rube-Goldberg"-esque, I think there's some real sensible reasons for this sort of regularization to make intuitive sense. The input-output Jacobian characterizes how much the output (i.e. the logits) are affected by small changes to the input. The Jacobian, reinterpreted as living in the input image space (as the authors do), is a map of which input pixels have the strongest effect on the output of the network. If the Jacobian image looks like the underlying input image -- in particular, highlighting the labeled object -- this indicates that changing those pixels will result in the largest change on the network output. (This should be clear when looking at Figure 4 of the paper.) On the other hand, adversarial noise by definition leaves the underlying object alone (so that a human isn't aware of the perturbation) and modifies other pixels. Models that fall for such adversarial noise will not have salient Jacobians.
This is an amusing original idea, and I think this paper probably should be accepted to ICLR -- though I don't hold that position very strongly. However, I think the most interesting point is idea of Jacobian saliency, which is from prior work (Tsipras et al., 2018) that I haven't read. Therefore, I'm not sure how significant this paper is on it's own. Regardless, I would have liked to see more discussion in the paper of why Jacobian saliency should confer robustness (as I tried to do in the paragraph above), with perhaps some additional experiments designed around understanding whether this intuition (or something similar) is actually correct. There's some discussion of the theory behind the method in section 3.1, but it's not very intuitive to the situation at hand (non-linear neural networks), and I don't find it particularly informative.
Finally, some effort is spent arguing that this method is more computational efficient than adversarial training -- I wonder if that's still true when the all the complexity of GAN training is taken into account or how to consider that point when part of the conclusion is that their method is best when it is also combined with some amount of adversarial training.


This paper considers a new variant of AMSGrad called Optimistic-AMSGrad. AMSGrad (Reddi et al. (2018)) is a popular adaptive gradient based optimization algorithm that is widely used in training deep neural networks. The new variant assumes that mini-batch gradients in consecutive iterations have some underlying structure, which makes the gradients sequentially predictable. By exploiting the predictability and some ideas from Optimistic Online learning, the proposed algorithm can accelerate the convergence and also enjoys a tighter regret bound. We evaluate Optimistic-AMSGrad and AMSGrad in terms of various performance measures (i.e., training loss, testing loss, and classification accuracy on training/testing data), which demonstrate that Optimistic-AMSGrad improves AMSGrad.

This paper proposes an online optimization method called Optimistic-AMSGrad, which combines two existing methods: (i) AMSGrad (Reddi et al 2018) and (ii) optimistic online learning where the prediction step is done with the extrapolation algorithm by Scieur et al 2016. The authors do a good job of presenting the method (by introducing the background in proper order), the paper seems self-contained and cites the relevant literature. The regret analysis of the proposed algorithm is provided, where the obtained regret can be smaller than AMSGrad depending on whether or not the guess of the gradient and the gradient are close.
In my opinion the boundedness assumption (footnote 2) is quite important here, and should be mentioned in the main text.
It is not clear how the different ways of accelerations combined in this method interact when the guess is not good. In other words, if the guess is not good this method could be slower then AMSGrad. Moreover, AMSGrad has stability property allowed from the ratio between 1st and 2nd moment estimate. In Optimistic-AMSGrad if m_t is bad, obtaining next w_{t+1} (line 9) would include ratio between outdated/bad 1st mom. estimate and new 2nd-moment estimate. In short, the method’s stability and outperformance might rely on the selection of the algorithm for gradient prediction.
In my understanding, extragradient has clear advantages in games, as if considering simple bilinear examples it is the only method that converges. However, for a single objective, its advantages are not clear to me (after reading the paper). Thus, I think it would be useful if the authors could provide comparison over *wall clock time* as well as long-run comparisons when the compared methods converge (it would be interesting to see if Optimistic-AMSGrad obtains better final train/test accuracy?). In many of the experiments where Opt-AMSGrad outperforms, the accuracy of the baseline still goes up--whereas the latter is computationally cheaper, so it is not clear from the provided results why a practitioner should use this method.
Moreover, the experimental results would be much more convincing if the authors do multiple runs using different seeds and present mean and standard deviation of the methods.
In the context of games, using more computationally demanding optimizers makes sense as training is unstable. In this case, after reading the paper, it is not clear to me what is the problem that the proposed method solves (or its advantages). Indeed, its advantage depends on how good the guess of g_{t+1} is. However, the extra-computation cost to obtain a good guess needs to be justified, or proven empirically that gets better performances faster (wall-clock time), or final ones.
In summary: (i)  the paper is well-presented and provides hyperparameter sensitivity results; (ii) the paper is very interesting, but (imo) it should leave clearer message why one should use this method; (iii) the proposed method has tighter regret, but only in some (data-dependent) cases and combines existing methods, limiting novelty. Hence, given the pros and cons, I am not confident recommending acceptance, and I think improved experimental results (error bars & wall-clock, see above) would make the results more significant.
--- Minor ---
- Alg.2: maybe add input hyperparameter r to optimistic-AMSGrad and a line between lines 8-9 that calls function which obtains the guess m_{t+1} (r should be passed to it). It would make it more clear that your algorithm has parameter r (sec. D.2)
- I could be wrong, but in my opinion, using \theta as first moment est. is slightly confusing, as it is normally used to denote parameters; similarly, the authors could use hat/prime on top of a variable to denote the ‘guess’ of that same variable, making it easier to follow.
- maybe add init of \theta_0 in alg1&2
- if I am correct amsgrad also does bias correction of the initial values of 1st and 2nd moment estimates; if that’s the case it could be useful adding a note that this is omitted for clarity if a reader implements it
- pg2: we just would like -> we would like


Graph representation learning embeds nodes in large graphs as low-dimensional vectors and benefit to many downstream applications. Most embedding frameworks, however, are inherently transductive and unable to generalize to unseen nodes or learn representations across different graphs. Inductive approaches, such as GraphSAGE, neglect different contexts of nodes and cannot learn node embeddings dually. In this paper, we present a context-aware unsupervised dual encoding framework, CADE, to generate representation of nodes by combining real-time neighborhood structure with neighbor-attentioned representation, and preserving extra memory of known nodes. Experimently, we exhibit that our approach is effective by comparing to state-of-the-art methods.

Strengths:
1. This paper presents a unified framework CADE for unsupervised graph representation learning, which combining the local neighbors and global information.
2. Based on GraphSAGE, the module DualENC can include positive pairs simultaneously, capturing the context-aware information.
3. In extensive experiments on Pubmed and PPI for link prediction, they demonstrate that their model beats the state-of-the-arts.
Drawbacks:
1. In this paper, the “Dual” refers to output adjacent nodes embeddings, where the adjacent nodes are collected by random walk. But the “dual” here is inconformity with its intrinsic definition and applications, according to the existing work “Dual-Primal Graph Convolutional Networks”. In other words, this paper lacks novelty, which seems incremental to the existing works. Specifically, the base-encoder is a variant of GraphSAGE plus a global embedding. The bi-attention is a variant of GAT.
2. The paper is not clear and structured. There are some confused logics and inaccurate interpretation. For example, in Introduction section, “Based on the hierarchical framework of GraphSAGE, GAT ….”. In Model section, there exists information redundancy and incoherence. For example, in section 3.2, equations 2-6 are reduplicative that have already appeared in algorithm 2.  Figure 1needs more information to help understand the model. To sum up, the connection and presentation of the four modules are not clear, which makes it difficult to read.
3. The experimental comparison methods and results are not complete. The baseline lacks the optimal method, “Deep Graph Infomax (DGI) ICLR 2019”, on unsupervised node classification. The results are not enough to support the claims (Only two benchmarks are compared).  Some important models on unsupervised graph representation learning are missing, e.g., Deep Graph Infomax(Veličković, 2018).


It is well-known that many machine learning models are susceptible to adversarial attacks, in which an attacker evades a classifier by making small perturbations to inputs. This paper discusses how industrial copyright detection tools, which serve a central role on the web, are susceptible to adversarial attacks. We discuss a range of copyright detection systems, and why they are particularly vulnerable to attacks. These vulnerabilities are especially apparent for neural network based systems. As proof of concept, we describe a well-known music identification method and implement this system in the form of a neural net. We then attack this system using simple gradient methods. Adversarial music created this way successfully fools industrial systems, including the AudioTag copyright detector and YouTube's Content ID system. Our goal is to raise awareness of the threats posed by adversarial examples in this space and to highlight the importance of hardening copyright detection systems to attacks.

[Summary] This paper proposed a black(white)-box attacking algorithms to attack industrial copyright detectors. Specifically, a simple un-targeted gradient-based method can successfully fool commercial system like YouTube and AudioTag. This paper has generalized the concept of “adversarial examples” to a new application area, which is indeed a very important one, to raise awareness for the potential threats.
There are a couple of reasons of why I like this paper: (1) copyright attack and especially defense (missing in this paper) is a more important application compared to others widely studied vision tasks like classification. To increase the copyright detectors' accuracy, companies must have invested a huge amount of money on deep learning based methods. As this paper suggests, these more accurate detectors also come with severe security problems, and such problems are rarely studied before. In addition, the experimental results in this paper show that two commercial systems can be easily compromised in a black-box manner. (2) This paper is clear in writing and easy to follow. I am not an expert in acoustic/audio recognition, but this paper shows that a not very complicated approach can work in real-world without too much domain/expert knowledge.
My rating for this paper is “3: weak reject”. I think it’s below the line for two reasons: technical novelty and experimental settings. I will elaborate these two points in the following sections.
As the author mentioned and generally believed, almost every machine learning model (black and white-box) can be compromised with adversarial examples, especially in un-targeted manner. “Copyright detector” is just another machine learning model regardless its important usage. The technical approach proposed in this paper is straightforward and limited. First, section 4.2 is very similar to Wang et al. (2003). What’s the biggest improvement or novelty here? All the filters are handcrafted and fixed as suggested by Eq.1 and Eq.2. Why don’t train a network or take a pre-trained DNNs to predict spectrogram?  The biggest advantage of deep learning (end2end trainable as feature extractor) is not utilized.  Second, the solution of Eq.5 is award. Usually we don’t use a softmax with temperature to estimate max, better ways should be tried like gumbel-softmax. Third, besides the optimization methods used for Eq.6, Carlini&Wagner is also widely used for optimizing this objective. What’s the reason that C&W is missed in this paper? Fourth, Eq.8 can be used for targeted attacks. However, this part is missing. Un-targeted attack is considered as easy especially in white-box cases. Targeted attack is a better and harder task for evaluation of algorithms. Fifth, the surrogate loss Eq.4 is basically a hinge loss between two quantized feature maps with mask \psi. It’s pretty standard to me and I am surprised this is the only loss proposed and studied.
The experiments in this paper is more like a “proof-of-concept” rather than “serious evaluation”. First problem is that the norm is used to evaluate the perturbation. Unlike the norm in image domains which can be visualized and easily understood, I don’t have a clear understanding of how “big” the perturbations are for audios. A cognitive study or something like a user study should be conducted. Another question related to this, the random noise is 3x bigger in terms of norm, does this make huge difference when listening to it? Are these two perturbations both very obvious or both unnoticeable? Second, it seems like a dataset is built but the stats are missing. How big is this dataset? How many songs are used to generate results in Tab.1, Tab.2, and Fig.3? What’s the attack accuracy on AudioTag and Youtube? Third, no baseline methods are compared to in this paper, not even an ablation study. The proposed two methods (default and remix) seem to perform similarly. I don’t think the best thing people can do previously to attack copyright system is just random noise. The authors could argue that there are no previous papers, but I think more realistic baselines should be studied.
One question, the inline equation at the bottom of Page.4  should be \phi(x) == maxpool(\phi(x)) right? And \psi(x) = ( \phi(x) == maxpool(\phi(x)) ) ?
Overall, I think this paper is not in good shape to be published even though the targeted problem has already become a big concern right now.


The goal of lifetime clustering is to develop an inductive model that maps subjects into K clusters according to their underlying (unobserved) lifetime distribution. We introduce a neural-network based lifetime clustering model that can find cluster assignments by directly maximizing the divergence between the empirical lifetime distributions of the clusters. Accordingly, we define a novel clustering loss function over the lifetime distributions (of entire clusters) based on a tight upper bound of the two-sample Kuiper test p-value. The resultant model is robust to the modeling issues associated with the unobservability of termination signals, and does not assume proportional hazards. Our results in real and synthetic datasets show significantly better lifetime clusters (as evaluated by C-index, Brier Score, Logrank score and adjusted Rand index) as compared to competing approaches.

The authors propose an approach to cluster subjects into K clusters according to their underlying, often unknown, life distribution. The loss function to the neural-network-based approach is based on a tight upper bound to the two-sample Kuiper test. Experiments are conducted on artificial and two real life datasets.
I enjoyed reading the paper, the authors propose an approach to address an important yet relatively under-explored problem in survival analysis.
It is not entirely clear how to handle the case when after the model is trained, H^(u) for a new subject u is larger than t_max when the model is trained. In such case, \Chi^(u) will be negative, thus what happens with \beta^(u)(W_2)?
When the termination signals are not available (friendster data) termination signals are set artificially when training and evaluation, thus, is the model learning artificially set termination signals and thus artificial survival functions? what is the point of doing this? Moreover, if the termination signals are artificial, what is the point of calculating C-Index and Integrated Brier score? I understand the motivation to compare to existing methods, it is really not clear how meaningful they are at evaluating performance.
In general, using C-Index and Brier scores in the context of the presented application may be misleading (or non-applicable) because these metrics are usually applied to survival analysis scenarios where one seeks to estimate likelihood of survival or time to event (over a usually infinite time horizon). Here, \beta^(u)(W_2) is a function of \Chi^(u) which is known (also not comparable to survival analysis), artificially tied to a pre-specified time-horizon t_max, and not dependent on covariates as in survival analysis. Further, SSC-Bair and SSC-Gaynor are clustering models, how are survival estimates obtained for these?
In practice, how is K selected? based on the friendster results for K=2,3,4,5 they all seem to produce distinct clusters, so which one should be used? This raises a question: how are the clusters, their members or the number of clusters informing the use case? In the MIMIC III experiment, where events are observed, the model finds two clusters without strikingly different survival functions (relative to friendster). Can one really consider S_2 as the high-risk group? Can one get higher risk clusters with larger K?
Minor:
- A_i^(u) is missing from the definition in the training data.
- In Section 3.2, if \xi^(u) is shared, why the superscript indicating that is subject specific?


Pose estimation is the task of finding the orientation of an object within an image with respect to a fixed frame of reference. Current classification and regression approaches to the task require large quantities of labelled data for their purposes. The amount of labelled data for pose estimation is relatively limited. With this in mind, we propose the use of Conditional Variational Autoencoders (CVAEs)  with circular latent representations to estimate the corresponding 2D rotations of an object. The method is capable of training with datasets that have an arbitrary amount of labelled images providing relatively similar performance for cases in which 10-20% of the labels for images is missing.

This paper tackles the task of rotation estimation in a setting where both labelled and unlabelled examples are available for training. It proposes to learn a generative model of images (a VAE), where the ‘code’ is factored into a latent vector z and the object rotation r.  As in training a VAE, an image encoder that predicts the distribution over (z, r) and the generator are jointly trained, but with additional supervision on the distribution over r for the labelled examples.
I think the overall idea of learning a disentangled generative model in a semi-supervised setting is simple and elegant, and could in principle help leverage unlabelled data. However,  I do have some concerns regarding the specific contributions of this work, and several reservations about the experiments reported, and would overall argue for rejection.
Concerns:
1) The central empirical result stated is that using this approach allows one to reduce amount of labelled data by 10-20 %. First, even if valid, this is not a very convincing reduction in the amount of supervision. However, I feel this claim is not well-established by the experiments:
1a)  The paper should report a baseline with only using the loss in eqn 3 and only training the encoder (using various fractions of training data) to predict the rotation i.e. purely discriminative training without training a generative model. The current plots of performance vs fraction of labelled data don't mean much until compared to a similar plot for this baseline. The current results don't really highlight the importance of training the generative model or using the unlabelled data.
1b) I think there are some inconsistencies in performances reported in Fig 2. I assume the test set is same despite different training data, because the paper states "All of the trained models are evaluated with respect to the complete test set". In this regard, I am puzzled why using 100% labelled data with 16 renders is significantly better than using 50% labelled data with 32 renders -- these should imply similar number of labelled examples, and more unlabelled ones in the former.
2) While the discussion points to this, the paper would really benefit from having results in a real setting, in particular as pose estimation is a field with a lot of prior methods that have been shown to work in these settings. The current results are all in a setup with synthetic, unoccluded data, without background variation, equidistant camera uniformly sampled along a circle. The central idea of using a generative model would be much more difficult to operationalize in a realistic setting where these simplifying assumptions are not made, and I'd only be convinced about the applicability of the approach by results in that setting. As a possible setup, one case use many imagenet images in conjunction with labelled examples in PASCAL3D+ to try this approach.
3) The overall approach maybe novel in context of pose estimation, but this idea of learning a disentangled generative model is not, and there are several papers which do so with varying amount of supervision e.g. see [1] below for similar ideas, and pointers. While some details here may vary, in context of these prior works, I'd view this paper as mostly applying well-established ideas to a new task.
--
In addition to the above, I have a question regarding the training/testing data:
Q: The dataset description only states data was randomly divided - was this random division at an image level, or model level i.e. could different renderings of the same model be in train and test set?
--
[1] Learning Disentangled Representations with Semi-Supervised Deep Generative Models, NIPS 2017. Siddharth et. al.


Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this problem is to learn domain-invariant embeddings for both domains. In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this is reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As a result, we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with or better than the best layer-dependent complexity tradeoff.

This paper studies the impact of embedding complexity on domain-invariant representations. By incorporating embedding complexity into the previous upper bound explicitly, the authors demonstrate the limitations of previous theories and algorithms. Based on their theoretical findings, the authors propose to control the embedding complexity with implicit regularization. Specifically, aligning source and target feature distributions in multiple layers controls both embedding complexity and domain discrepancy. The proposed algorithm can achieve similar performance as DANN with manual selection of embedding depth.
By noting that the hypothesis space can be decomposed in to the feature extractor and the classifier, the authors propose to address the domain discrepancy separately. D_H\DeltaH is termed latent divergence, which the algorithm attempts to minimize. D_G\DeltaG is treated as embedding complexity, which is the intrinsic property of the feature extractor. Thus, domain-invariant representations should seek a proper tradeoff between those two terms.
The paper is well-written and the contributions are stated clearly. The exploration on the layer division is really insightful.
However, I have several concerns:
1.	The proposed upper bound is insightful, but it has several limitations. Compared to the version applied to the feature space in equation (3), the proposed upper bound is looser. The embedding complexity terms includes two encoders, which are deep neural networks in practice, thus it can be excessively large. As the authors point out, in equation (3), the embedding complexity is not addressed explicitly, but it is implicit in the adaptability \lambda in a more reasonable way. Previous works [1], [2], [3] have already taken them into consideration. Proposition 5 is a direct application of proposition 1 in [1].
2.	On the claim of implicit regularization. By applying domain adversarial training to multiple layers, the authors claim that the encoder in higher layers is implicitly restricted. However, they do not validate this regularization effect. Is the embedding complexity controlled? Theoretical analysis or experimental results would be helpful.
3.	The proposed MDM method seems to be incremental. [4] has probed into the effect of multi-layer adaptation strategy. Besides, applying domain adversarial training to many layers leads to more computational cost and may slow down training significantly.
[1]Fredrik D Johansson, Rajesh Ranganath, and David Sontag. Support and invertibility in domain- invariant representations. arXiv preprint arXiv:1903.03448, 2019.
[2]Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon. On learning invariant representation for domain adaptation. arXiv preprint arXiv:1901.09453, 2019.
[3] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pp. 4013–4022, 2019.
[4] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, volume 37, pp. 97–105, 2015.