The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019;Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

CONTRIBUTIONS:
I use (unqualified) “self-attention” to refer to attention of tokens in a sequence to other tokens in the same sequence, as described by [some corrected version of] Eq (1) and the paragraph following it (citing Bahdanau et al. 2015). This contrasts with “Transformer self-attention” and “cross-sequence attention”.
C1. Self-attention is gating. (Prop. 4.1, Sec. 4)
C2. Gating cannot provide explanations in the way that attention is alleged to do.
C3. Single input sequence models deploy only self-attention, so by C1-C2, attention cannot be used for explanation of these models; attention in two input sequence models is not limited to self-attention and is not equivalent to gating, so attention can provide explanations in these models.
C4. Outputs are sensitive to alteration of attention weights for two-sequence but not one-sequence models. (Tables 2-3 vs. 1; Figs. 2-3)
C5. Human judgments (“manual evaluation”) of the intuitive importance, for the output, of items with highest attention weights show that such intuitive importance is found for both one- and two-sequence models. (Fig. 5)
RATING: Reject
REASONS FOR RATING (SUMMARY). Because the modeling experiments closely follow previous work, the primary contribution rests on the account provided of why explanation-by-attention works only sometimes --- on the basis of the Proposition identifying attention with gating. But the reasoning and the math as presented are problematic. There is a worthwhile contribution from the human judgment experiment, but this is not sufficient to overcome the weakness with the main argument of the paper. (I also have reasons to question whether one- vs two-sequence inputs is the right distinction that needs to be accounted for.)
REVIEW
C1. If this point were made in plain English, ‘attention weights and gating units both multiply activation values’, I would say, “yes, obviously”. But the point is stated as a Proposition, with a Proof, so the math given should be correct. I don’t see a way to make it correct, and that shouldn’t be the job of the reader anyway. There are several errors in the dimensions of the matrices which make the equations incoherent. Eq. (1) contains W*h, a matrix product, where the dimensions stated are W in R^{d x d’} and h in R{T x m}; these cannot be multiplied. This unclarity about matrix dimensions propagates into Prop. 4.1. In the definition of g(X), we have WX + b, where b is presumably a vector. Addition then requires that WX also be a vector, but X is stated to be in R^{n x d}, so WX cannot be a vector. Whether WX + b is actually a vector or a matrix, g(X) = c^T tanh(WX + b) is not a matrix: it is either a scalar or a vector. But this can’t be. The definition of h uses the elementwise product, which requires that both arguments have the same dimensions, so g(X) must have the same dimensions as f(X). We’re told f is the identity function, so f(X), like X, must be a matrix. Furthermore, the statement of Prop. 4.1 says that self-attention can be interpreted as *a* gating unit. By the standard definition of ‘unit’, this should mean that self-attention is a scalar.
Throughout the paper, we are never told what kind of “RNN” is being assumed. If the RNN unit contains gates, as in an LSTM or a GRU, I can imagine that the intention is for Prop. 4.1 to say that (*) “the effect of self-attention can be reproduced without attention by adjusting the weights in the gating mechanism already present in the RNN”, so that attention doesn’t increase the capacity of the model. But what I see in the paper does not convince me that (*) is true. (Because of the kind of global normalization required by the softmax, I actually suspect it is not.)
C2. I don’t see why (formally or intuitively) gating is not a legitimate candidate for explaining the behavior of networks containing gates; I would assume just the opposite, actually. How can it *not* be part of a satisfactory explanation? And why should changing a name from “attention” to “gating” have any bearing on whether it (whatever it is called) can potentially serve for explanation of network behavior?
C3. Leaving the formalism aside, I don’t see intuitively why, whatever an analysis of self-attention might entail about explanation, the same implication shouldn’t apply to straightforward (not Rocktaeschel) attention when two sequences are present. Why can’t we just treat the concatenation of the input sequences as a single input sequence, as standardly done for example for the Transformer? If the formal content of Prop. 4.1 were clear, perhaps this could be justified, but it is simply asserted without justification in the proof that “the same reduction does not hold in the case of pair sequence”.
C4. Claims C1-C3 attempt to give an account for why various tests of the explanatory role of attention turn out positive for two-sequence but not one-sequence tasks, a pattern previously reported and verified with new results in the present paper. I fear however that one- vs two- is not the correct empirical generalization about attention that one should try to account for. Messing about with attention weights would not be expected to alter outputs if the output is determined by the input considered as a bag of words. And there is a troubling confound in the tasks compared: the one-sequence tasks are sentiment and topic classification, where a BOW model should do very well – and I suspect that is the real reason why these tasks don’t show strong sensitivity to attention weight distribution. But the two-sequence tasks are NLI and QA, where (ideally) BOW models should not do nearly so well: paying attention to the right tokens should be important. The same is true of translation. So the confound in the tasks examined leaves it undetermined whether the crucial factor to account for is one- vs two-sequence or BOW-friendly vs BOW-unfriendly tasks.
C5. Put together, the human judgments and the sensitivity-to-altering-attention-weights seem to indicate that attention tends always to be allocated to intuitively important tokens, and that matters for the output of the two-sequence models but not for the one-sequence models. This is what we’d expect if attention is always being allocated appropriately, but for BOW-friendly tasks that doesn’t make much difference.


We propose a multi-agent learning approach for designing crowdsourcing contests and all-pay auctions. Prizes in contests incentivise contestants to expend effort on their entries, with different prize allocations resulting in different incentives and bidding behaviors. In contrast to auctions designed manually by economists, our method searches the possible design space using a simulation of the multi-agent learning process, and can thus handle settings where a game-theoretic equilibrium analysis is not tractable. Our method simulates agent learning in contests and evaluates the utility of the resulting outcome for the auctioneer. Given a large contest design space, we assess through simulation many possible contest designs within the space, and fit a neural network to predict outcomes for previously untested contest designs. Finally, we apply mirror descent to optimize the design so as to achieve more desirable outcomes. Our empirical analysis shows our approach closely matches the optimal outcomes in settings where the equilibrium is known, and can produce high quality designs in settings where the equilibrium strategies are not solvable analytically.

1. Summary
The authors employ a multi-agent learning approach for learning how to set payoffs optimally for crowdsourcing contests
and auctions. Optimality means e.g. incentive alignment (the principal problem) between the principal (e.g. the organizer) and participants (e.g. bidders), assuming e.g. that participants can be strategic about their behavior. In this work the principal uses ReLU-log utility.
First, the authors use fictitious play and multi-agent RL to train agents on a distribution of payoffs. Then, a neural net is fitted to the samples (payyoffs, expected principal utility), and finally iteratively attempts to improve the payoffs using mirror ascent within the convex set of admissable payoffs.
The authors compare the payoffs with theoretically known solutions and in situations where the optimal solution is not known.
3, 4-agent all-pay auction (Nash eq known).
Same as above, but with noise added to bids (Nash eq not known).
The authors analyze in some detail how the principal's utility and bidder ranking behave as the participants' bids change.
1. Decision (accept or reject) with one or two key reasons for this choice.
Reject. Although the high-level approach is interesting (use learning to design auctions for cases where no theoretical solution is known), the actual experimental results and methodological improvement over e.g. Dutting 2017 are weak. The authors only consider 3, 4-agent auctions. There are no other learned baselines (e.g., constrained optimization without neural nets) that the authors could consider.
3. Supporting arguments
See above.
4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
M-EMA and M-EMD: ascent and descent? in Algo 1, 2?
---
I've read the rebuttal, but still lean towards reject. The scope/analysis of the experiments (e.g. auction type), still seems limited, even though both agents and mechanism are adaptive.


Meta-learning for few-shot learning involves a meta-learner that acquires shared knowledge from a set of prior tasks to improve the performance of a base-learner on new tasks with a small amount of data. Kernels are commonly used in machine learning due to their strong nonlinear learning capacity, which have not yet been fully investigated in the meta-learning scenario for few-shot learning. In this work, we explore kernel approximation with random Fourier features in the meta-learning framework for few-shot learning. We propose learning adapative kernels by meta variational random features (MetaVRF), which is formulated as a variational inference problem. To explore shared knowledge across diverse tasks, our MetaVRF deploys an LSTM inference network to generate informative features, which can establish kernels of highly representational power with low spectral sampling rates, while also being able to quickly adapt to specific tasks for improved performance. We evaluate MetaVRF on a variety of few-shot learning tasks for both regression and classification. Experimental results demonstrate that our MetaVRF can deliver much better or competitive performance than recent meta-learning algorithms.

This paper proposes a meta-learning framework for learning adaptive kernels using a meta-learner. For representing kernels, the paper learns a variational posterior for the kernel features, by maximizing the Evidence lower Bound. Furthermore, to plug the kernel learning into the meta-learning framework, they let the variational feature posterior to condition on the current support set for adapting and to use a modified LSTM network for accumulating information. Empirically, they compare the proposed MetaVRF with multiple baselines in the standard fewshot classification benchmarks and demonstrate superior performance. They also illustrate that their adaptively-learnt Fourier feature outperforms the standard variational Fourier features.
Strengths,
1, The idea of learning kernels in meta-learning is interesting. In fact, learning a kernel is equivalent to learning a distance between objects. If a reasonable distance between objects can be learnt, using the corresponding kernel should be able to achieve superior performance even if the kernel doesn't adapt in each episode.
2, The proposed method achieves competitive performances. In particular, Figure 5 shows how the performance changes when the test-shot and test-way are varied. It seems surprising that the MetaVRF achieves >90% accuracy for 100-way test when trained on only 5-way 5-shot.
Weakness,
1, The notations in the paper are not well presented. (a) In eq(2), the formula $alpha^t=\Lambda(\Phi^t(x), y)$ is not exact, cuz $\alpha^t$ should depend on the whole support set $S^t$ while $(x,y)$ is only one instance in $S^t$.  (b) In eq(11), the variational posterior $q(w | S^t, w^{1:t-1})$ is not exact either. Because $w^{1:t-1}$ are random variables, they cannot be observed and cannot be conditioned on. Similar issues also exist in the caption of Figure 1.
2, The paper doesn't introduce what is the likelihood $log p(y| x, S, w)$. It is unclear how the kernel regression is adopted in classification.
3, The meta-prior $p(w| x, S)$ depends on the feature of the query point, which doesn't seem to be a common practice in variational inference. It would be beneficial if the authors could explain this and probably validate it empirically.
4, The motivations of the modified LSTM should be clarified more. Does the paper remove h_t in LSTM for removing short-term memory ? Why $\hat{c}_t$ only depends on $e^t$ instead of $[e^t, c^{t-1}]$ ?
5, The paper compares with multiple competitive baselines. However, the settings for the baselines should be better presented. For example, it is strange that the numbers of SNAIL are different with the numbers in their paper. And SNAIL on Omniglot is not reported. Furthermore, another competitive method TADAM (Oreshkin et. al., 2019) should also be compared with.


In this paper, we propose a novel kind of kernel, random forest kernel, to enhance the empirical performance of MMD GAN. Different from common forests with deterministic routings, a probabilistic routing variant is used in our innovated random-forest kernel, which is possible to merge with the CNN frameworks. Our proposed random-forest kernel has the following advantages: From the perspective of random forest, the output of GAN discriminator can be viewed as feature inputs to the forest, where each tree gets access to merely a fraction of the features, and thus the entire forest benefits from ensemble learning. In the aspect of kernel method, random-forest kernel is proved to be characteristic, and therefore suitable for the MMD structure. Besides, being an asymmetric kernel, our random-forest kernel is much more flexible, in terms of capturing the differences between distributions. Sharing the advantages of CNN, kernel method, and ensemble learning, our random-forest kernel based MMD GAN obtains desirable empirical performances on CIFAR-10, CelebA and LSUN bedroom data sets. Furthermore, for the sake of completeness, we also put forward comprehensive theoretical analysis to support our experimental results.

Overview:
The paper propose an MMD GAN extension via using Random forest Kernel.  Instead of using Gaussian kernel on the top of the learned embeddings from the discriminator, it combines existing deep forests kernels. The theory of being differentiable is carefully studied (to prove zero measure) and the experiments are well conducted.
1.  Some important  references are missing.  One very related paper is
* Li et al., Implicit Kernel Learning,  AISTATS 2019.
That paper is using the same idea to learn to manipulate the random features on the top of the learned embedding.  The main difference between it and the proposed algorithm is they use MLP parameterization instead of the tree-based model.    Also, the deep forest model can be treated as a sparse neural network, does it have more advantage over Li et al., (2019)? given they use simple dense MLP.   Please at least discuss the similarity and difference in the rebuttal and update the draft correspondingly.  I would even encourage the author to empirically compare with it in the camera ready version.  It would be interesting to see which parameterization is better in this space.
There are also other recent MMD GAN extensions should be cited in the discussion, such as
* On gradient regularizers for MMD GANs.
2. For the theory part, based on Binkowski (2018), the gradients for the generator parameters should be biased. Could you discuss it with Theorem 2?
3. For most MMD GAN results, one important property in Li et al., (2017),  Arbel et al., (2018) and Li et al., (2019) is weak* topology.  Does the proposed Random Forest MMD GAN also has that property? In Li et al., (2019), they need some condition to ensure that, how's case in the proposed algorithm?


Graph embedding techniques have been increasingly deployed in a multitude of different applications that involve learning on non-Euclidean data. However, existing graph embedding models either fail to incorporate node attribute information during training or suffer from node attribute noise, which compromises the accuracy. Moreover, very few of them scale to large graphs due to their high computational complexity and memory usage. In this paper we propose GraphZoom, a multi-level framework for improving both accuracy and scalability of unsupervised graph embedding algorithms. GraphZoom first performs graph fusion to generate a new graph that effectively encodes the topology of the original graph and the node attribute information. This fused graph is then repeatedly coarsened into a much smaller graph by merging nodes with high spectral similarities. GraphZoom allows any existing embedding methods to be applied to the coarsened graph, before it progressively refine the embeddings obtained at the coarsest level to increasingly finer graphs. We have evaluated our approach on a number of popular graph datasets for both transductive and inductive tasks. Our experiments show that GraphZoom increases the classification accuracy and significantly reduces the run time compared to state-of-the-art unsupervised embedding methods.

Summary: This paper proposes GraphZoom, a framework for augmenting unsupervised graph embedding methods by (a) fusing feature information into the graph topology, (b) learning embeddings on a coarsened graph, and (c) refining the coarsened embeddings to obtain embeddings for the original graph nodes. In particular, a nearest neighbor graph over node features is computed and this adjacency matrix is linearly combined with the original adjacency matrix to obtain a graph with feature information "fused in". The graph is then coarsened using a spectral approach, embeddings are learned on the coarsened graph (via any strategy), and the embeddings are then refined back to the original nodes (again using a spectral approach). The authors take care to heed the advice of Maehara et al. and remove high-frequency information from the features.
Assessment: Overall, this is a borderline contribution with some interesting motivation, original ideas, and sound derivations. However, the primary limitation of this work is the empirical comparison. First, the empirical comparison includes DeepWalk and GraphSAGE as the two base models, and while these are reasonable models, they are known to no longer be state of the art in this area (e.g., see https://arxiv.org/pdf/1809.10341.pdf). It would be more appropriate to include a more recent and better performing method (e.g., DGI; linked previously), as the reported numbers are very far from state-of-the-art. In addition---and perhaps a more concerning issue---is that seems that a randomly initialized GCN can obtain similar or superior performance compared to the numbers reported in this work (again, see the DGI paper linked above). While it is possible that GraphZoom+DGI or GraphZoom+[some other more recent method] could achieve stronger results, the fact that the current results seem to be below performance of a randomly initialized GCN is a major issue. Stronger empirical results with better baselines and base models would drastically improve the paper.
As another point regarding the empirical results, the datasets used are known to be problematic (e.g., see https://arxiv.org/abs/1811.05868). If these datasets are used, then multiple random splits should be employed and more robust summary statistics should be reported.
Regarding the fusion step, there were also two points that should be addressed in the paper:
1) It seems that this fusion setup is assuming that the network exhibits homophily (i.e., it assumes that nearby nodes have similar features). This is common in many networks (e.g., the benchmarks that are analyzed) but not always the case. Some commentary on when (if ever) this fusion process might *not* be appropriate would improve the paper.
2) The authors state the they use the coarsening process to compute the nearest neighbor graph in order to avoid the quadratic time complexity. However, there are numerous well-established approaches to deal with this issues (e.g., locality sensitive hashing). Why was one of these standard approaches not employed?
Reasons to accept:
- Original and well-motivated idea
- Clearly written paper
Reasons to reject:
- Problematic empirical evaluation (e.g., lacking recent baselines)
- Several performance numbers appear to be below random GCN baseline performance
- General applicability of the approach (e.g., to non-homophilous networks) is not clear


Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.

This paper proposes a novel one-shot-pruning algorithm which improves the training of sparse networks by maximizing the norm of the gradient at initialization. The utility of training sparse neural networks and shortcomings of dense-to-sparse algorithms like Pruning, LotteryTicket are nicely motivated at introduction. The pruning criterion is motivated by the first order approximation of the change in the gradient norm when a single connection is removed, though the results show that removing many connections together with GraSP increases the total gradient norm therefore allowing the loss to decrease faster. Experiments suggest employing such pruning algorithm improves final performance over two baselines: random and SNIP.
Though I find the proposed method intriguing and well motivated, experiments section of the paper misses some important sparse training baselines and needs some improvement. I am willing to increase my score given my concerns/questions below are addressed.
(1) The paper doesn't mention some important prior work on the topic. Since the paper focuses on end-to-end sparse training, the following sparse training methods needs to be considered and compared with:
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science [Mocanu, 2018]
- Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization [Mostafa, 2019]
- Deep Rewiring: Training very sparse deep networks [Bellec, 2017]
- There is also few recent work submitted to ICLR2020: https://openreview.net/forum?id=SJlbGJrtDB, https://openreview.net/forum?id=ryg7vA4tPB, https://openreview.net/forum?id=ByeSYa4KPS
(2) Pruning baselines can be improved. I am not convinced that they represent the best achievable sparse training results. I would recommend method proposed by `Prune or Not to Prune` as a strong baseline. You can also check `State of Sparsity` paper to obtain some competitive pruning results.
(3) It's great that the authors are aware of the importance of having experiments on larger datasets. Though, I found the results reported on Imagenet to be limited. Is there a reason why Imagenet-2012 results are missing pruning baselines? I think having other reported pruning results here along with performance of other sparse training methods (SET, DSR) would be useful. Most of these numbers should be readily available in the papers mentioned above, but I guess it is always better to run them using the same settings.
(4) To demonstrate the usefulness of the pruning criteria proposed, it would be nice to do some simple ablations. Some suggestions: (1) Remove weights that would *decrease* the gradient norm most (2) Do random pruning while preserving exact per layer sparsity fractions. (3) sweep over batch size used to calculate the importance scores and evaluate final accuracies or the initial gradient norm. The second experiment would help identifying whether the gains are due to better allocation of sparsities across layers or due to increased gradient norm. Looking at Figure-4 and seeing the per layer sparsities are different, It is not clear to me which one is the underlying reason for improved performance.
(5) (Page 8 / Table 5) Do you aggregate all accuracies in Table 5 using different batch sizes and initialization methods? If, so I am not sure what the intended message here is, since it is difficult to infer how these hyper-parameters affect the result. Do you sweep different batch sizes for estimating importance of units, too? It would be nice to see whether the two batch sizes interact with each other and/or how increased batch size affects the quality of pruned networks.
Some minor comments:
(a) (Page 1) I found the motivation very intriguing. Though the statement `Recently, F&C (2019) shed light on this...` seems a bit off, given that LT can't find solutions as well as the pruning solution in most practical (larger datasets and architectures) settings. Therefore I would be better to pose this as an `open problem`.
(b) (end of page-1) `However, connection sensitivity is sub-optimal as a criterion because the gradient of each weight might change dramatically after pruning due to complicated interactions between weights`. I think this is still the case for GraSP. Since the criterion it uses assumes independence (i.e. what if we remove a single weight?). It would be nice to see some ablations on this. Does `K=number of weights removed` affect the norm of the sparsified networks?
(c) (Figure 1) I find the comparative illustration between SNIP and GraSP very useful. Though, the architecture presented seems a bit artificial (i.e. I am not aware of any architecture with single hidden layer and a single output unit). I think the same motivation can be made by removing the top unit (therefore having 6-4-1 units) and removing all incoming connections for the output unit until a single connection remains. Then SNIP would remove that single connection whereas GraSP would remove one of the connections in the previous layer.
(d) (Section 2.1) `In contrast, Hessian based algorithms...` Though it is a structured pruning algorithm It might be nice to include the following work, https://arxiv.org/abs/1611.06440.
(e) (Section 2.1) Previous work needs following citations:  [Bellec, 2017], [Mocanu, 2018] and [Mostafa, 2019]
(f) (Section 2.2) Why the initial dynamics affect the final performance? One explanation given in the paper is through recent work on NTK and this is great. Though training settings used at `Lee et al (2019a)` and in the paper are a bit different. Usage of MSE, small datasets, etc… So it might be nice to point out differences.
(g) (Section 3) At $D = {(x_i, y_i)}_{i=1}^n$, `n`->`N`
(h) (Page 4) `Preserving the loss value motivated several…` -> `motivated by several…`
I think it is better to use existing terminology whenever available.I think using `One-shot pruning` instead of `Foresight pruning` would be a better choice and would prevent confusion.
(j) (Page 5) `However, it has been observed that different weights are highly coupled …` This has been observed much earlier, too: like in Hassibi, 1993.
(k) (Page 7) Last sentence `and thus hopefully..`: needs to be fixed.
(l) (Page 8) The whole page needs some proof-reading. Some of them: (a) `SNIP and GraSP. We present...` probably connect with comma (b) `aims for preserving` -> `aims to preserve` (c) `In contrast, SNIP are more` `are`->`is` (d) `for ablation study` -> `as an ablation study`...
(m) Is there a specific reason why VGG networks are preferred for experiments? I don't think they are relevant to any practical application anymore and they are massively over-parameterized for tasks in hand. Specifically for Cifar-10. I think focusing on more recent networks and larger datasets would increase the impact of the work.


Deep reinforcement learning has succeeded in sophisticated games such as Atari, Go, etc. Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations. This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations. DPFRL encodes a differentiable particle filter with learned transition and observation models in a neural network, which enables reasoning with partial observations over multiple time steps. While a standard particle filter relies on a generative observation model, DPFRL learns a discriminatively parameterized model trained specifically for decision making. We show that the discriminative parameterization results in significantly improved performance, especially for tasks with complex visual observations, as it circumvents the difficulty of modeling complex observations that are irrelevant to decision making. Experiments show that DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this work. Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment.

Update: my concerns have been addressed and I have updated the score to 8
****
This paper introduces 3 neat ideas for training deep reinforcement learning (DRL) agents with state variables so that they can handle partially observed environments:
1) model the latent state variable as a belief distribution, using a collection of weighted hidden states, like in the particle filter (PF), with an explicit belief update of each particle, calculation of the weight using an observation function, and a differentiable re-weighting function to get the new belief distribution,
2) base the policy on the whole set of particles, by quantifying that set using its mean as well as a collection of K learnable moments (specifically, K Moment Generating Functions, each one corresponding to a dot product between the moment variable and the hidden state of the particle),
3) instead of generating the observations, take again the idea from PF which is to measure the agreement between the current observation o_t and the i-th particle state variable h_t^i, via a learnable discriminative function.
From what I understand, the only gradients in the model come from the usual 3 RL losses, and the observation functions in the discriminative PF are trained because they weigh the particles.
The model, trained using Advantage Actor Critic (A2C) works well on the (contrived, more on that later) "flickering Natural" Atari RL environment as well as on the Habitat navigation challenge, outperforming both the GRU-based deep RL agent and the Deep Variational RL based agent that uses variational sequence auto-encoders (and extra gradients from the observation function...). The ablation analysis confirms the advantages of the 3 ideas introduced in the paper.
The paper is a very well written and the experiments are very well executed. I believe that the idea is novel. I gave this paper only a weak accept because of unclear explanation and of several missed opportunities:
* The observation function f_{obs}(h_t^i, o_t) is insufficiently explained. I understood it was trained using discriminative training. Does it mean that different observations o_t are used, and if so, how many? Or is the observation o_t the current observation of the agent, but only the h_t^i change? In which case, what makes it discriminative? Isn't there a posterior collapse, with all particles ending up bearing the same state? Does the function f_{obs} input o_t or u(o_t), where u is the convolutional network?
* These questions could be easily answered with pseudocode in the appendix.
* In section 3.1, what is the relationship between p_t(i) and f_{obs}(h_t^i, o_t)?
* Particle filters in navigation enable to store the history of the observations of the mobile robot, accumulating the laser range scans and matching them to the observations. At the end, one can visualise the map stored in a given particle, as well as visualise the point cloud of the particle coordinates and show the trajectories of these particles. Here the particles contain the hidden states of the agent. Could you similarly to traditional PF, visualise the position of the agent by matching the point cloud {{h_t^i}_i}_t to a set of observations o_k taken from the whole environment, and plotting a 2D map of weights coming from function f_{obs}(h_t, o_k) evaluated over all k?
* In the discussion, can you comment on the relationship between Monte-Carlo Tree Search in RL agents (sampling different trajectories) vs. here (sampling different states)?
* While I understand the need to use that environment for the sake of comparison to DVRL, the Atari + flickering + natural images dataset is very artificial and contrived. I would be interested in seeing more analysis of the discriminative PF RL algorithm on navigation tasks, given that that's what PF were designed for.
Some missing references:
* Early references on DRL for navigation:
Zhu et al (2016) "Target-driven visual navigation in indoor scenes using deep reinforcement learning"
Lample & Chaplot (2016) "Playing FPS games with deep reinforcement learning"
Mirowski et al (2016) "Learning to navigate in complex environments"


Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity. NN quantization is usually formulated as a constrained optimization problem and optimized via a modified version of gradient descent. In this work, by interpreting the continuous parameters (unconstrained) as the dual of the quantized ones, we introduce a Mirror Descent (MD) framework (Bubeck (2015)) for NN quantization. Specifically, we provide conditions on the projections (i.e., mapping from continuous to quantized ones) which would enable us to derive valid mirror maps and in turn the respective MD updates. Furthermore, we discuss a numerically stable implementation of MD by storing an additional set of auxiliary dual variables (unconstrained). This update is strikingly analogous to the popular Straight Through Estimator (STE) based method which is typically viewed as a "trick" to avoid vanishing gradients issue but here we show that it is an implementation method for MD for certain projections. Our experiments on standard classification datasets (CIFAR-10/100, TinyImageNet) with convolutional and residual architectures show that our MD variants obtain fully-quantized networks with accuracies very close to the floating-point networks.

The paper proposes to use the mirror descent algorithm for the binary network. The key point is Theorem 3.1, which enables the mirror map. The paper is easy to read and follow, and the main contributions are clearly stated.
However, I suggest a weak rejection of this paper. The reasons are
Q1. As Review #3, it is better for authors to provide more theoretical analysis, which better includes the nonconvex objective function and the effect of annealing.
Q2. It is not clear to me, why mirror descent is better than proximal gradient descent, i.e., proxQuant, in this application. The authors repeatedly claim "MD allows gradient descent to be performed on a more general non-Euclidean space". This cannot be told by Table 1, which is just overall performance. So, it is better to empirically show this point by an ablation study.
Q3. Since the technical contributions are not enough, I expect more experimental comparisons.
- Could the authors perform experiments on ImageNet?
- While VGG and ResNet are taken as a protocol for experimental comparison, it is better to do an extra comparison with STOA networks. VGG and ResNet are too old and easy to be compressed, compression these networks are of little practical values. EfficientNet [1], Mobilenets [2], and Shufflenet [3] can be good ones. The paper will be more convincing with these methods.
[1]. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
[2]. Mobilenets: Efficient convolutional neural networks for mobile vision applications
[3]. Shufflenet: An extremely efficient convolutional neural network for mobile devices


Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community. To address it and to establish baselines and a common evaluation protocol in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form. Specifically, we investigate in-domain representation learning to develop generic remote sensing representations and explore which characteristics are important for a dataset to be a good source for remote sensing representation learning. The established baselines achieve state-of-the-art performance on these datasets using in-domain representations.

### Summary
This paper discusses transfer learning in remote sensing image-classification tasks.
The key finding is: while transfer learning from ImageNet dataset for remote sensing, performance can be slightly improved if the model weights are fine-tuned on an auxiliary remote sensing dataset (hence in-domain transfer).

### Strengths
- The paper is well motivated for the need of good representation learning for remote sensing, and is written well making it easy to follow.
- It compiles a nice list of five datasets available for remote sensing, along with their characteristics. Also, the list of common properties of such datasets (e.g. rotation invariant) is good for data pre-processing for engineering applications in remote sensing.
- The paper provides baseline accuracies on five important datasets in remote sensing.

### Weaknesses
Apart from the contributions of compilation of different datasets which are already present, and classification results on them, there is not much novelty in the paper. There are several major weaknesses:

1. **About datasets**: The paper emphasizes a lot on the lack of standard datasets and metrics in remote sensing. However, the contribution is limited to compiling existing datasets and defining a 60-20-20 train-validation-test split on them. There is no new data collection, no insight in how these existing diverse datasets could be combined together, or why this particular definition of split is used. Hence, the reviewer cannot count this as a contribution.
2. **About representation learning**: Since the source domain (ImageNet), in-domain source domain and the target domain all solve classification tasks, a better term to use would be *transfer learning* instead of representation learning. The reviewer recommends to demonstrate how these learned "representations" are used for other downstream tasks or otherwise show some other representation learning methods (semi-supervised or unsupervised), which is not same as the downstream task. Hence, it would be inappopriate to sell it as representation learning, in its current form.
3. **About in-domain learning**: Since the in-domain transfer is also done by fine-tuning an existing model which was trained on ImageNet, it is expected to do somewhat better than Imagenet only. The paper just clarifies this fact in practice, and the performance gains are also statistically insignificant. Again, this is not a significant contribution.
4. **About hyperparameter sweeping**: The minor improvements observed due to in-domain transfer is also unreliable, since very hyperparameters are tried across datasets. E.g. only 2 values of learning rates (0.1 and 0.01) are tested. This could also explain why BigEarthNet and So2Sat had poor training accuracies. The reviewer also disagrees with using only 1k samples for selecting best dataset to transfer from - explained in minor weaknesses.
5. **About experiments**: In-domain transfer is tested by fine-tuning on one dataset and then again fine-tuning on another dataset. It was observed that the dataset RESISC-45 does the best in this scenario. The reviewer disagrees with paper's hypothesis about noisy labels resulting in this artifact. The reviewer suspects that this is because RESISC-45 contains labels which are most appropriate for classification transfer in other datasets (and also it is very balanced). Can the authors comment on this? If another method of representation learning than classification is employed, then it is very possible that other larger datasets result in better representations. The comparison across datasets currently is strange, since all it tells is which dataset has most similar classification labels to other datasets on average. To remove this problem in comparison, the representation learning task and testing task should be different.
6. **About conclusions and contributions**: The experimental setup in this paper is insufficient to claim that it investigates *what characteristics should a dataset have to be a good source for remote sensing representation learning*. Transfer from ImageNet works because it contains a large number of classes, along with data diversity and its sheer size. This cannot be said for the datasets analyzed in this paper, since these factors are not analyzed individually, and the representation learning method is not standardized across all datasets (classification is not standardized since different datasets have different classes), making the comparison unfair.

#### Minor weaknesses:
- Table 4: Since existing BigEarthNet result contains precision/recall, that is what the authors should provide for their method.
- The best dataset to transfer from is evaluated only on 1k samples, which is too small. The accuracy changes ~25% to ~70% accuracy for BigEarthNet when going from 1k samples to full dataset. Hence the trend is unpredictable, and some other dataset could have potentially performed better on full dataset.
- Remove blank page at the end

### Suggestions to Improve
For this to be a comprehensive survey on in-domain representation learning in remote sensing, the reviewer recommends to add:

1. Standardized representation learning methods to compare across datasets.
2. Results on more downstream tasks (other than classification; examples mentioned in the paper introduction) to evaluate representation learning.

The reviewer understands these could be out of scope of this paper. However in its current state, the contributions are not significant enough.


Min-max formulations have attracted great attention in the ML community due to the rise of deep generative models and adversarial methods, while understanding the dynamics of gradient algorithms for solving such formulations has remained a grand challenge. As a first step, we restrict to bilinear zero-sum games and give a systematic analysis of popular gradient updates, for both simultaneous and alternating versions. We provide exact conditions for their convergence and find the optimal parameter setup and convergence rates. In particular, our results offer formal evidence that alternating updates converge "better" than simultaneous ones.

*Summary*
This paper studies the convergence of multiple methods (Gradient, extragradient, optimistic and momentum) on a bilinear minmax game. More precisely, this paper uses spectral condition to study the difference between simultaneous (Jacobi) and alternating (Gau\ss-Seidel) updates. The analysis is based on Schur theorem and give necessary and sufficient condition for convergence.

*Decision*
This paper tries to study a phenomenon that has known a recent surge of interest due to the numerous practical minmax application: the impact of alternating updates in minmax optimization. This problem is challenging because most of the theoretical analysis techniques have been developed to analyse simultaneous updates.
I think that this paper has treated well the related work and underlines well its contributions.
Even if the main contribution are theoretical the authors of this paper provide some experiments to confirm that the theory provides some meaningful insights.
However, I have the concern that the study is still limited to bilinear example and thus it is not clear how to apply it to non-bilinear objective (even locally because since the Jacobian has pure imaginary eigenvalues, even locally the Jacobian may have eigenvalues with negative real part).
Moreover this work do not provides significantly better (factor 8 improvement in the constants) convergence rate that the one provided on the literature (Tseng 1995, Mokthari et al 2019, Gidel et al. 2019) to solve bilinear games. The contributions is more about proving new (interesting) analysis tools and showing a better robustness of GS updates with respect to hyperparameter tunning.
To me, it is an accept.
*Questions*
- In Theorem 4.1 and 4.2 you use the word ‘optimal exponent’. In what sense these algorithms are optimal ? Usually the sense of optimal is when you achieve the lower bound of convergence. Is it the case here ?
- Right after theorem 2.3 you claim that GS updates simply leads to a shift of index for $L_i$ setting $L_{k+1}=0$ but it seems to me that then $\lambda L_1$ is missing in the sum. Is it just an unfortunate oversight? Does it affect the proofs of you main theorems ? These results only marginally (improve by a factor 8) improve upon Moktari et al. (2019).

*Remarks*
- To me the J and GS conditions in Theorem 3.2, 3.3 and 3.4 are more Lemmas than Theorems: They are condition that are hard to interpret and they have small interest if one cannot conclude on simple conditions (such as the ones you actually provide in the Theorem). My point is that providing such complicated conditions without condition is only half solve the problem of convergence.  In that sense, in the discussion of Theorem 3.4, you are a bit unfair with Gidel et al (2019) since unlike them you do not provide any convergence rate. On one side theorem 3.4 gives an interesting characterization for the convergence (that leads to the fact that at least one of the beat has to be negative) but if it does not lead to any rate it is a less interesting result.
- In Table 1 and 2, $\alpha,\beta_1$ and $\beta_2$ have not been introduced. It is thus hard to get the most of them.
- Theorem 3.1 has been stated in previous works (like for instance Gidel et al 2019). You should not claim it as a contribution.
=== After rebuttal ===
I've read the authors's response.
I agree with the concerns raised by Reviewer 2 regarding the experimental part and whether or not such theoretical study (far from the practical aspect) are of interest to the ICLR community.
I think that that paper would be more suited for a theoretical venue such as AISTATS (or ICML) but I also think that this work remains of interest to the ICLR community.
I am not convinced by the fact that this analysis could be generalized beyond bilinear game with a local analysis  because even an arbitrarily small perturbation can transform a bilinear example into a non-stable equilibrium (with no hope of local convergence), take for instance
$$ \min_x \max_y -\epsilon \|x\|^2 +  x^\top A y + \epsilon \|y\|^2$$


Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks. This leads to a poor performance on old tasks -a phenomenon framed as catastrophic forgetting. While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization. Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network. This requires to calculate the Hessian around a mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian. In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task. Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work.

1. Summary:
The paper considers neural network training in the continual learning setting -- data arrive sequentially and we can not revisit past data. The paper proposes an approximate Laplace’s method, in which the Hessian the log likelihood of the data is approximated by some form of Hessian-vector project (? - I will get to this question mark below). The paper considers some benchmark continual learning datasets and compares the proposed approach to EWC and Kronecker-factored online Laplace. The performance of the proposed approach is similar to that of EWC and worse than Kronecker-factored Laplace in most cases. Another sales pitch that the paper brings up a lot is the low space complexity, however this benefit has not been fully demonstrated, given the small-scale network/experiments.
2. Opinion and rationales
I’m leaning towards “strong reject” as I think the presentation needs another round of polishing and that the technical contributions need to be clarified / unpacked. I explain my thinking below.
a. The presentation/explanation/flow are not clear.
The abstract does not read well. For example: “This requires to calculate the Hessian around a mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.” This sentence makes it sound like current approaches are tractable, so what this paper is trying to address? The technical summary is also not precise, the Hessian-free methods used in the paper is to compute Hessian-vector products, not the actual Hessian.
The introduction motivates the continual learning problem using generalisation of neural networks leading to the need for multi-task learning; however multi-task learning is not scalable given the large number of tasks and thus we need to learn sequentially. However, I find this motivation not clear: if multi-task learning and its scalability issue are the reasons why we need continual learning, with the scale of the experiments considered in the paper, wouldn’t it always more beneficial to use multi-task learning instead of continual learning?
The prior work section is also not clear, in my opinion. The paper starts out by describing EWC as Bayesian updates and cites MacKay (1992), then talks about the Kronecker-factored Laplace approximation as “address this shortcoming by adopting the Bayesian online learning approach”, as if these methods are very different while in fact, these methods are some variants of the Laplace approximation, with different ways to approximate the Hessian. The issues described in section 2.2 “two problems that stem from eq 1” are not very clear, for example, “without storing the information from all previous tasks there is no easy solution to update the posterior” (?). I would follow the presentation/explanation in Ritter et al (2018), Huszar (2018) [a note on the quadratic penalty of EWC] and section 5 of the variational continual learning paper (Nguyen et al 2018) to provide a more succinct connection between these methods.
The connections between this work and MAML in section 3 is not clear to me. The continual learning and meta learning settings are also quite different.
b. The technical contribution is not clear and if correct, if of limited novelty.
What is not clear from reading section 3 is what quantity is being approximated, at what point a Hessian-vector product appears and thus we can use Hessian-free methods to approximate it. The paper talks about flat loss surface and sampling a small subset of the Hessian -- I’m not sure I understand these connections. In eq 11, the paper replaces the Hessian values with results of the Hessian-vector-product approximations -- this seems very odd to me, especially in terms of semantics and units, Hessian and hessian-vector-products are two very different things. Again, it is perhaps just me not understanding what is being approximated in the first place. The technical contribution of this paper is thus limited: using Hessian-free methods to approximate Hessian-vector products in the continual learning context.
c. The performance of the proposed method is not super exciting. Pragmatically speaking, it is not clear why practitioners should be using this in the near future given Kronecker-factored Laplace works and scales well in practice and there are a plethora of other recent methods (e.g. VCL) that are also developed from the Bayesian principle and work much better than EWC.
3. Minor details:
a. In eq 1, the denominator should be p(D_{t+1} | D_{1:t}).
b. Figs 1 and 2, I would use the same colour scheme throughout to be consistent.


Saliency maps have been used to support explanations of deep reinforcement learning (RL) agent behavior over temporally extended sequences. However, their use in the community indicates that the explanations derived from saliency maps are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which saliency maps represent semantics of RL environments. We evaluate three types of saliency maps using Atari games, a common benchmark for deep RL. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are an exploratory tool not an explanatory tool.

[score raised from weak accept to accept due to rebuttal/improvements]
Summary
The paper investigates the practice of using pixel-level saliency maps in deep RL to “explain” agent behavior in terms of semantics of the scene. The main problem, according to the paper, is that pixel-level saliency maps often correlate with semantic objects, however turning these correlations into explanations would require counterfactual analysis / interventions, which are almost never performed in practice. The paper highlights this issue with an extensive literature survey, and proposes a simple method to formulate “explanations” found via saliency maps into falsifiable hypotheses. Three experiments show how to apply the methodology in practice - in all three cases pixel-level correlations cannot be easily mapped to semantic-level explanations that hold (counterfactual) validation.
Contributions
The paper nicely summarizes the main contributions, namely: (i) a literature survey on pixel-saliency methods in deep RL and their use to “explain” agent behavior, (ii) a detailed description of the problem with the latter and a proposal to mitigate the main issues, and (iii) three experimental case-studies to illustrate the problem further and show how the proposed method can help.
Quality, Clarity, Novelty, Impact
The paper addresses a highly important issue in the field of interpretable deep learning. The main message is that a lack of scientific rigor, namely stating falsifiable hypotheses and validation of claimed hypotheses, can easily lead to misinterpretation of deep RL systems. This is a somewhat disenchanting message, but I personally think it is important to ensure that this message is heard in the field of interpretable ML in particular, and in the wider deep learning community in general. It is tempting to give simple answers to complex problems, and while I think saliency maps will play a large role in interpreting deep network decisions, I am also convinced that we need causal explanations, which salience maps (currently) cannot provide on a semantic level. The paper is well written and clear, the literature survey is quite extensive and valuable. The experimental results are nice, however they currently crucially lack quantitative statements that back up the qualitative results (see improvements below). While the latter must be included for publication, I am fairly confident that this can be rectified during the rebuttal phase and therefore (tentatively) vote for acceptance.
Improvements
a) Mandatory for publication! Back the results-plots up by numbers! In particular: visually estimating densities / correlations from scatter plots is often impossible and misleading - while the plots are nice to have, the claims regarding Figure 5, 8, 9 (b) and (c) must be backed up by reporting actual correlations / statistical tests. For instance, it is impossible to judge visually whether there’s any trend in 5 (c). Please report correlations for 5, 8, 9 (b) and perform suitable statistical tests for measuring increase/decrease in correlation for 5, 8, 9 (c).
Similarly, please report an appropriate metric to quantitatively judge the difference between the curves in 4, 6, 7 (c). It’s fine to include tables reporting the quantitative results in the appendix, they don’t necessarily have to be in the main paper.
b) Experimental details. Please report the details required to reproduce the experiments. In particular, what was the precise architecture for A2C and the hyper-parameter settings (particularly since the reference that is cited is not a paper, but a GitHub repo). For the figures, please report how saliency was measured exactly (was there a bounding-box around saliency/enemies? What was its size? Were intensities somehow normalized, were distances to enemies measured between centers of bounding-boxes, …?)
c) (Optional). It would be nice to see an example where the method is used but the original hypothesis is not rejected (i.e. there’s now stronger evidence for the original hypothesis due to the counterfactual analysis). I understand that this is beyond the scope of the rebuttal, and feel free to completely ignore this.
Major Comments
I) Please state whether the paper was written with feed-forward deep RL agents only in mind, or whether the paper is intended to also include recurrent deep RL agents (it would also be helpful to know whether the experiments used a feed-forward, or a recurrent version of A2C). While I think that many aspects carry over from feedforward architectures to recurrent ones, I personally think that some issues with counterfactual analysis could become more intricate with recurrent agents. For instance, on page 6, the described invariance in the first paragraph under ‘Counterfactual Evaluation of Claims’ is fine for feed-forward agents, but could be debatable with recurrent agents.  If you agree, please make this distinction clear in the paper (where appropriate) or state that the paper only applies to feedforward agents. If you disagree please indicate this during the rebuttal discussion.
II) Page 6, just above Sec. 5: “Since the learned policies should be semantically invariant under manipulations of the RL environment...”. I agree that they should ideally be invariant, for the semantic interventions to make sense, but please comment on whether this is a trivial assumption, how this assumption could (in principle) be verified and the potential consequences of this assumption being violated. I personally think that there’s a fair chance that the semantic space carved up by the agent (that potentially overfits a task/family of tasks) might be quite different from the semantic space given by the latent factors of the environment. This mismatch and its potential interference with the method should be discussed as a current shortcoming.
Minor Comments
I)  A potential subtlety (which I don’t expect you to resolve/discuss in the paper) is that feed-forward agents in an MDP environment can behave like recurrent agents by offloading memory into the environment. E.g. a breakout agent could “memorize” that it is in “tunnel-digging mode” by moving the paddle by a few pixels - this could then potentially shift it’s saliency away from the actual tunnel to the corresponding pixels around the paddle. Such cases might be very hard to interpret via saliency maps or interventional analysis, but I acknowledge that this is perhaps a more exotic case, given the current state of interpretable deep RL. Just a thought for future work perhaps...


Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge. Both spatial and temporal contextual information play a critical role in the domain of visual recognition. Here we investigate spatial constraints (what image features provide contextual information and where they are located), and temporal constraints (when different contextual cues matter) for visual recognition. The task is to reason about the scene context and infer what a target object hidden behind a flap is in a natural image. To tackle this problem, we first describe an online human psychophysics experiment recording active sampling via mouse clicks in lift-the-flap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy. As a proof of the usefulness of these clicking patterns and visual features, we extend a state-of-the-art recurrent model capable of attending to salient context regions, dynamically integrating useful information, making inferences, and predicting class label for the target object over multiple clicks. The proposed model achieves human-level contextual reasoning accuracy, shares human-like sampling behavior and learns interpretable features for contextual reasoning.

The paper is about context reasoning in the visual recognition. They designed a task called 'lift-the-flap', where the human or the model are given an image with one of the object blacked out. The image is provided in a relative low resolution, and the subjects could choose to click on area to reveal a local window of high-res image. The subject then need to answer what is the object in the blacked out region after a few clicks (ranging from 1 to 8 clicks). The authors has collected human data using mTurk, and also trained a model (named ClickNet) to perform the task. ClickNet has a LSTM memory which can carry the information of previous clicks and use spatial attention mechanism to decide the next click location (just use the argmax of the attention weight.) I find the paper is clearly-written and quite easy to follow. I think this is an interesting paper comparing human attentional performance to a trained computation model (without any human-imposed prior), showing that they behave similarly. However, I am not sure it would be of general interest to the ICLR audience. This may be more suitable for cognitive science conference/journal in my opinion. I have a few suggestions below:
1. In formula 2, the attention weight (before softmax) is  " e_ti = A_h * h_{t−1} + A_a * a_ti " I am surprised there is no term that is h_{t-1} * a_ti, which do the content-based reading that is common in memory model like (DNC) or in language processing (transformer.) Can the author comment on this decision?
2. In result 5.1, the authors titled: "WHAT: IMPORTANCE SAMPLING VIA ATTENTION AND PRIOR INFORMATION". I am not sure I get where is the importance sampling appear in this paragraph. Importance sampling is a specific term, which refers to estimating properties of a particular distribution, while only having samples generated from a different distribution than the distribution of interest. I did not see how that is related to this paragraph. Or, I may have missed it, and would like the authors to elaborate.
3. I could not follow well the discussion in section 5.5 WHEN: ROLE OF RECURRENT CONNECTIONS. Based on the title of the section, I would expect to discuss why the recurrence of LSTM is important. Though, the paragraph is mainly talking about other control models like SVM and HMM, so I don't quite follow it. Also, it is not clear to me the word 'When' is a good choice. I am convinced that the LSTM helps the model carrying information across click, but it is not clearly shown the sequential order is critical (say click location 1, then location 2, then location 3 compared to location 2, location 3, then location 1 matters) Maybe it does? But, it is now shown in the results.
4. The comparison of where the human versus the model click (in Figure 6) is impressive. I wonder if the authors could go even one step further that is to see if the temporal order of the clicks are similar in human vs. model. It may not be the case. However, if it is true, that could make an even stronger case that the model has a similar way to decide attentional location compared to human subjects.
Minor:
1. It is impressive that the authors have considered a wide range of control models. Though, for a conference paper with limited page limit, I don't think it is necessary to describe all these models in the main text ( for the SVM, HMM, and CRF models.)


We introduce a novel neural network-based partial differential equations solver for forward and inverse problems. The solver is grid free, mesh free and shape free, and the solution is approximated by a neural network. We employ an unsupervised approach such that the input to the network is a points set in an arbitrary domain, and the output is the set of the corresponding function values. The network is trained to minimize deviations of the learned function from the strong PDE solution and satisfy the boundary conditions. The resulting solution in turn is an explicit smooth differentiable function with a known analytical form. Unlike other numerical methods such as finite differences and finite elements, the derivatives of the desired function can be analytically calculated to any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is a unified formulation of both forward and inverse problems where the optimized loss function consists of few elements: fidelity terms of L 2 and L ∞ norms that unlike previous methods promote a strong solution. Robust boundary conditions constraints and additional regularizers are included as well. This setting is flexible in the sense that regularizers can be tailored to specific problems. We demonstrate our method on several free shape 2D second order systems with application to Electrical Impedance Tomography (EIT), diffusion and wave equations.

In this work, the authors propose an approach to solve forward and inverse problems in partial differential equations (PDEs) using neural networks. In particular, they consider a specific type of second-order differential operator combined with Dirichlet boundary conditions, and suggest how neural networks with suitable training objectives can be used to solve both types of problems. For a specific elliptic system relevant for Electrical Impedance Tomography (EIT), they then present numerical results obtained with their method and compare those to high-resolution finite element solutions and previous work.
The paper addresses a very general and important problem with wide ranging applications. People have solved PDEs using spectral, finite difference, finite volume or finite element methods for decades and there is a huge body of literate on this subject. The neural network based approach proposed in this paper seems general and simple with encouraging experimental results. However, there are several important points missing in the current paper based on which I would recommend rejecting it. However, I would be willing to increase the score if these points were addressed in sufficient detail.
Major comments:
1) Choice of problems
There are many important problems in the literature that belong to the general type of PDE considered in this work (e.g. in electrostatics). To appreciate and understand the merits and limitations of the proposed approach better, it would be highly necessary to apply it to a wider range of problems, including ones with known analytical solution.
2) Convergence tests
Convergence tests are an important part that is missing in the current paper. In particular, how does the error in the neural-network solution decay with respect to the number of random points used to train the network? Plots comparing a suitable error norm versus the training dataset size would be very informative. Furthermore, what are the conditions, if any, to guarantee convergence to the exact solution? The authors mention in the last paragraph of the Discussion that a more rigorous analysis will be published elsewhere. However, as error analysis is such an integral part of the study, I think it needs to be addressed, at least to some extent, already in the current paper.
3) Related work
I don’t think the current work is put in sufficient contrast with existing work. Several related papers are mentioned in the introduction and the experimental section includes one other method (GDM method of Sirignano & Spiliopoulos) for comparison. However, a more thorough discussion on how the current approach complements existing literature on neural network based PDE solvers would be in order.
Minor comments:
i) There are quite a few typos and grammar mistakes in the paper that need to be fixed. To give a few examples:
‘the natural structure presents’ -> ‘the natural structure present’
‘Shrödinger’ -> ‘Schrödinger’
‘approximated by neural network’ -> ‘approximated by a neural network’
‘on circular and’  -> ‘on a circular and’
‘forces the equation’ -> ‘enforces the equation’
etc...
ii) Eq. (7): There is a parenthesis missing on the LHS.
iii) The authors show many plots in Figs 3-9 but don’t comment much on the results in the main text. A lot of the results presented could therefore go into an appendix. I would find it better to present fewer results/plots and discuss those in more detail.
iv) It would be good to define the Nabla operator in Eq. 5 for completeness.
v) Please put references to equations and papers in parentheses or, at least, separate them otherwise from the sentence, to avoid confusion. For example:
- the reference to ‘Evans (2010)’ below Eq. 5, or
- the list of dangling references before the last paragraph on page 3,
- the reference to Sirignano & Spiliopoulos in the caption of Table 1,
etc...
vi) Please expand the captions in Fig. 3 to make the figure meaningful as a stand-alone, without having to read the entire text to understand what 'phantom' means. The same applies to all other captions.
vii) Perhaps it would be clearer to use ‘row’ instead of ‘line’ when referring to the results in the table.
viii) In Sec. 7 ‘s’ in ‘Ns’ should be a subscript.
ix) Explain what the acronym PSNR means.
x) Please elaborate on why you take the top k values in Eq. (7). What happens if you take more/less?
xi) First paragraph in Discussion: I am not sure the robustness w.r.t. the hyperparameters is really so surprising given that you apply the solver to very similar problems. If you could show that the same hyperparameters worked on a completely different problem, that would be much more interesting.
xii) Please add references for the first paragraph of the introduction.
*********************************************************
I increased my rating based on the revisions.
*********************************************************


Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task. In this work, we study the underlying potential causes for this specialization by measuring the similarity between representations trained on related, but distinct tasks. We use the recently proposed projection weighted Canonical Correlation Analysis (PWCCA)to examine the task dependence of visual representations learned across different embodied navigation tasks. Surprisingly, we find that slight differences in task have no measurable effect on the visual representation for both SqueezeNet and ResNet architectures. We then empirically demonstrate that visual representations learned on one task can be effectively transferred to a different task. Interestingly, we show that if the tasks constrain the agent to spatially disjoint parts of the environment, differences in representation emerge for SqueezeNet models but less-so for ResNets, suggesting that ResNets feature inductive biases which encourage more task-agnostic representations, even in the context of spatially separated tasks. We generalize our analysis to examine permutations of an environment and find, surprisingly, permutations of an environment also do not influence the visual representation. Our analysis provides insight on the overfitting of representations in RL and provides suggestions of how to design tasks that induce task-agnostic representations.

This article studies the similarities between the learned representations for different tasks when trained using reinforcement learning algorithms. The ultimate question that this study tries to answer is an interesting one. Namely, how much can representations learned by training on one task be beneficial for learning other tasks? A high interdependence between the representations can lead to a more successful transfer of knowledge between tasks.
The authors are further interested in studying the properties that influence this relationship, which depends on the elements of training as well as attributes of the tasks themselves.
However, I believe that the results are not strong enough to support the claims that are stated in the paper and the limited scope of the environments tested does not make a convincing case that the results will be generalizable much beyond these scenarios. Therefore, in the current state, I think this paper should be rejected.
Firstly, the paper states that:
> ".. if the distance between models trained on different tasks is the same as that between models trained on the same task, representations are task-agnostic."
This seems to be a key argument in the paper. However, I believe that this argument is based on the assumption that representations learned for a single task are indeed highly similar. I think this assumption requires some sort of support as reinforcement learning algorithms are known to be highly inconsistent even in reaching similar solutions. Therefore, one cannot take for granted that the learned representations would be similar in any way.
Second, the authors claim in Section 5 that the random splits have little impact on the learned representation, but in Section 6 claim that the spatially disjoint splits have a noticeable impact on the representations. Without any measure of the impact, looking at figures 1b and 3a, I'm not convinced that this distinction is so obvious. The situation is worse when looking at the figures in the appendix, namely figures A3 and A5. If someone were to swap these two figures, my untrained eye would not be able to tell the difference.
I suggest that the authors spend more time explaining their reasoning as to why these results are significant enough to support the claims. Also, the text should be improved if it is to be accepted. There exist many problems ranging from small typos (simlate -> simulate, reuse-ability -> reusability, and "?." -> "?") to sentences that need to be reworked. Some figure legends/captions can also be improved to include more information, such as explaining what exactly the shaded regions represent (I'm guessing one standard deviation from the mean over some unknown replicates), or in Fig 3b making clear whether "A -> B" is done with "New Policy" or "Fine-tuned Policy". I am also curious as to why only one of these scenarios was experimented with in sections 6 and 7.


Stochastic Gradient Descent (SGD) with Nesterov's momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance. In this work, we propose Amortized Nesterov's Momentum, which is a special variant of Nesterov's momentum. Compared with Nesterov's momentum, our new momentum has more robust iterates and higher efficiency. Our empirical results show that it achieves faster early convergence and comparable final generalization performance with little-to-no tuning. Just like Nesterov's method, the new schemes are also proved optimal in general convex setting. Our analysis sheds light on the understanding of the new variant.

%% Post Author Response comments %%
Thank you for your detailed response/revision.
1 - Introducing “m-times” larger momentum: Somehow, this is not a particularly intuitive statement or one that reflects clearly in a theoretical bound. Since we are getting to issues surrounding the use of momentum with stochastic optimization, I would like to make a note that the performance of these algorithms more broadly aren't quite sketched out for their use in broader stochastic optimization. In particular, despite broad use in practice, it is unclear if standard variants of Nesterov acceleration/Heavy Ball method achieve "acceleration" in stochastic optimization. See for e.g., the work of Kidambi et al ICLR 2018 (“On the insufficiency of existing momentum schemes for stochastic optimization”) - where, the argument was that these methods were designed for deterministic optimization (where we get exact gradients) - in fact, that paper empirically as well as theoretically shows that these schemes do not offer acceleration in a precise sense compared to specialized algorithms for stochastic optimization. It is unclear if the proposed algorithms can offer a similar improvement over SGD in a provable sense, even for the specific examples described in their paper.
2 - The point about theory (just as you mention) is that it doesn’t directly apply towards the simulations, nor, do they improve on already known algorithms - so I am unable to see the point that these results present broader implications that can guide practice.
3 - The response doesn’t address the fact that for the theory bounds presented in the paper to hold (even in the convex settings described), one requires knowledge of parameters that are not known a-priori, and are often fairly difficult to estimate. So the performance of the algorithm in practice may quite significantly be away from the bounds described in the paper.
While I appreciate the points and revision made by the authors, I still believe the paper requires some rethinking to present their results (and this includes more detailed comparisons to existing works) in order to make a case towards broader practical applications.
%%. %%
This paper considers robustness issues faced by Nesterov’s Acceleration used with mini-batch stochastic gradients for training Deep Models. In particular, the paper proposes amortized momentum, an algorithm that offers a way to handle these issues. The paper in general is well written and easy to follow.
The paper proposes algorithms AM-SGD1 and AM-SGD2 and presents extensive results regarding their complexity analysis on convex problems and their performance when training neural networks. The algorithms require storing one more model’s worth of storage compared to standard momentum based methods (which can be viewed as a drawback in certain cases).
Comments:
[1] I am concerned about the motivation behind this paper - which, according to the paper is that Nesterov’s accelerated gradient method with stochastic gradients has huge initial fluctuations. The issue with regards to more fluctuations of the initial performance is natural given how aggressive these accelerated methods work. As long as this is not a reason/cause for worse terminal performance (which doesn’t seem to be the case), I am unable to see why large initial fluctuations are concerning.
[2] Theory: The theory bounds for this problem setting do not appear to improve over known bounds in the literature. As a side note, the work of Hu et al. “Accelerated Gradient Methods for Stochastic Optimization and Online Learning” is highly related to this paper’s theoretical aspects, setup and bounds. Furthermore, this bounded variance noise model for stochastic gradients, while being theoretically useful (and important), is often very detached from practice (as this implies that the domain is bounded and we perform projections of iterates whenever they go outside the set - such aspects hardly reflect on practical SGD implementations). Using this as a means to reason about robustness of the proposed algorithm (for e.g. remarks for theorem 1a. and in conclusions) appears to be a big leap that may lead to potentially misleading conclusions.
[3] In order to run the algorithm to achieve the theoretical bounds claimed (in theorems 1 and 2), it appears that the stepsize \alpha_s depends on unknown quantities such as initial distance to opt, noise variance etc.
[4] The claim in page 2 about comparing SGD and M-SGD says that the stepsize in deterministic and stochastic optimization is constrained to be O(1/L) is rather misleading. In realistic practical implementation of SGD with a multiplicative noise oracle, one really has to use a much smaller stepsize than 1/L. This in a sense leads back to point[2] about the unrealistic nature of bounded variance assumptions for understanding SGD based methods used in the context of Machine Learning. They are better suited for understanding stochastic methods in black-box optimization (as opposed to considering Machine Learning problems).
My take is that even if the authors justify novelty in terms of theory results (which, to my knowledge is limited compared to existing literature), rewriting the paper by considering its theoretical merit and presenting empirical results (even as considered in this paper) of this algorithm (without attempting to make very strong connections to explain issues experienced in non-convex training of neural networks, since the theory works in vastly different settings under restrictive assumptions) can be appreciated by appropriate sections of audience (both in theory as well as optimization for deep learning communities).


A number of techniques for interpretability have been presented for deep learning in computer vision, typically with the goal of understanding what the networks have actually learned underneath a given classification decision. However, when it comes to deep video architectures, interpretability is still in its infancy and we do not yet have a clear concept of how to decode spatiotemporal features. In this paper, we present a study comparing how 3D convolutional networks and convolutional LSTM networks, respectively, learn features across temporally dependent frames. This is the first comparison of two video models that both convolve to learn spatial features but that have principally different methods of modeling time. Additionally, we extend the concept of meaningful perturbation introduced by Fong & Vedaldi (2017) to the temporal dimension to search for the most meaningful part of a sequence for a classification decision.

This paper presents a paradigm for generating saliency maps for video models, specifically, I3D (3D CNN) and C-LSTM. It extends Fong & Vedaldi, 2017 to generate a temporal mask and introduces two types of "meaningful perturbations" for videos: freezing and reversing frames; they use Grad-CAM (with no modifications) for generating spatial masks. The problem is well-motivated, as saliency maps have been extensively studied for image classification models, but rarely for video classification. Quantitatively, they demonstrate that frame-reversal is meaningful for the Something-something dataset but less for KTH because those actions rely more on spatial information than temporal (i.e., running, clapping). Qualitatively, they show their spatial and temporal masks on both datasets and suggest the a few insights:
* I3D's Grad-CAM visualizations show a center, default bias * I3D is less sensitive to the reverse perturbation * I3D temporal masks are typically shorter.
I currently rate this paper as a weak reject (though closer to borderline) paper for the following reasons (all of which can be improved in rebuttal):
1. Quality of technique
While the temporal masks are novel and qualitatively "make sense" (though this is subjective), the generation of the spatial masks is not novel, is unconnected to the temporal mask generation, and often doesn't make sense because of lack of temporal smoothness / cohesion, particularly for C-LSTM, where the visualizations when the mask is on appear quite "jumpy" (see Fig 2, Seq 1). It would be great to see more innovation on the temporal mask generation to address some of these issues (one natural approach that comes to mind would be learn spatial masks as done in Fong & Vedaldi, 2017, possibly with a temporal smoothness term between spatial masks and possibly combining temporal + spatial masks for freezing operation, i.e., only freeze spatial pixels) -- that said, I realize that this may be out of scope for a rebuttal.
2. Lack of support for qualitative claims
The paper makes 3 claims based on qualitative examples shown (see above bullets in summary); these claims could be easily substantiated qualitatively (by evaluating over the dataset or a subset of it).
3. Lack of discussion on limitations/benefits of technique + how to use/interpret technique
* There are no baseline comparisons for the proposed temporal mask generation. natural ones would be visualizing saliency methods (i.e., gradient, SmoothGrad, occlusion [Zeiler & Fergus, 2014], RISE [BMVC 2018], for instance, as examples of a few easy-to-implement, representative methods for backprop and perturbation methods) w.r.t. to temporal dimensions and then thresholding and applying those baseline temporal masks and demonstrating that the proposed temporal mask generation is.
* There's no discussion (or experiments) on the benefits & limitations of their approach; this is important as we've seen from papers like Mahendran & Vedaldi, ECCV 2016, Adebayo et al., NeurIPS 2018 and Kindermans et al., arXiv 2017 that some saliency methods fail to meet basic desirata (i.e., specificity to output class and model weights, etc.). Ideally, the authors would show that their temporal mask generation meets some desired criteria (as well as compare with baseline methods) to justify their approach.
* One limitation of Fong & Vedaldi 2017 is the difficulty of finding global optimum for the different hyper-parameter terms (Fong et al, ICCV 2019 addresses this, which might be of interest to the authors). There's no discussion about whether this problem persists for this work. Also -- it'd be interesting to see whether the reverse loss function (i.e., maximizing class score) yields similar results.
* More discussion can be added about how to interpret results / use the technique (i.e., what is this technique useful/not useful for? what do results mean?). For instance, the claim that I3D temporal mask is shorter suggests a complex phenomenon -- that the necessary temporal evidence is smaller. This is a bit surprising to me, as I3D performs better overall, so I would have expected it to encode more redundancy (this can be checked by exploring the classes in which I3D does perform better) -- however, this interpretation differs from that of the authors ("This is especially visible in the temporal mask of Sequence #3, where it is active specifically").
In addition to responding to the above with relevant text + preliminary experimental results, I also had the following questions/asks:
* For misclassified classes in Fig 2, are you optimizing w.r.t. the top predicted class or the ground truth class? Do they differ substantially when optimizing for different output classes?
* What were the lambda hyperparameters from Eq. 1 (and how were they chosen)? Are these relatively stable or are their instances of technique failure due to the difficulty in balancing these terms?
* Show Table 1 on only the classes that were focused on (i.e., the ones w comparable performance between the two models); I'm wondering if the impact of reversal on I3D is less than that on C-LSTM, as claimed by the authors (it's hard to tell when their baseline performance is different)


Graph neural networks have shown promising results on representing and analyzing diverse graph-structured data such as social, citation, and protein interaction networks. Existing approaches commonly suffer from the oversmoothing issue, regardless of whether policies are edge-based or node-based for neighborhood aggregation. Most methods also focus on transductive scenarios for fixed graphs, leading to poor generalization performance for unseen graphs. To address these issues, we propose a new graph neural network model that considers both edge-based neighborhood relationships and node-based entity features, i.e. Graph Entities with Step Mixture via random walk (GESM). GESM employs a mixture of various steps through random walk to alleviate the oversmoothing problem and attention to use node information explicitly. These two mechanisms allow for a weighted neighborhood aggregation which considers the properties of entities and relations. With intensive experiments, we show that the proposed GESM achieves state-of-the-art or comparable performances on four benchmark graph datasets comprising transductive and inductive learning tasks. Furthermore, we empirically demonstrate the significance of considering global information. The source code will be publicly available in the near future.

This paper presents two models, namely GSM and GESM, to tackle the problem of transductive and inductive node classification. GSM is operating on asymmetric transition matrices and works by stacking propagation layers of different locality, where the final prediction is based on all propagation steps (JK concatenation style). GESM builds upon GSM and introduces a multi-headed attention layer applied on the initial feature matrix to guide the propagation layers. The models are evaluated on four common benchmark datasets and achieve state-of-the-art performance, especially when the training label rate is reduced.
Overall, the paper is well-written and its presentation is mostly clear and comprehensible (see below). The quantitative evaluation looks good to me, especially since an ablation study shows the contributions of all of the proposed features.
However, there are a few weak points which should explain my overall score:
1. The proposed GSM model is not new and only re-uses building blocks from the related work. [1] shows that removing non-linearities is an effective procedure for node classification. [2] investigates the massively stacking of propagations. The procedure of feature concatenation from different locality has been studied in [3]. Applying asymmetric normalization is a standard aggregation scheme for GNNs, e.g., in [4].
2. The GESM model is not fully understandable since it is missing a formal description for computing $\alpha$. It is only said that $\alpha$ is computed using the concatenation of features from the central node and its neighbors. Can you elaborate how exactly you compute $\alpha$, especially since the concatenation of neighboring features results in a non-permutation invariant architecture? In addition, in contrast to the reported results in Tables 3 and 4, Figure 4 indicates that the benefits of GESM are negligible.
3. The final prediction layer with weight matrix W_1 operates on all propagation layers, resulting in a parameter complexity of $O(s  h c)$, where $c$ is the number of classes. With $s=30$ and $h=512$, this results in 15.360  c parameters (!!!), whereas GCN [5] only uses 16  c parameters. Hence, I do not think it is fair to promote your model as efficient as vanilla GCN. In addition, the final matrix multiplication results in a computational complexity of $O(n  s^2  h^2  c)$ which does not nearly match your reported complexity.
Furthermore, I do wonder why your model is not heavily overfitting with such an amount of parameters. For example, this is the reason [3] does evaluate its model on a larger training split instead of a smaller one.
4. As your work is quite similar to [1, 2], it would be beneficial to also include the respective results of those methods in Tables 2 and 3. In addition, their differences and similarities should be discussed in detail.
5. Since the used benchmark datasets are already reasonably explored, authors are advised to include evaluation on other datasets as well, e.g., from [6].
6. The transition matrix $P$ is missing self-loops to match with the results of Figure 1. Since you already define $P$ analogously to $\hat{\tilde{A}}$, you should focus on one notation for consistency reasons.
[1] Wu et al.: Simplifying Graph Convolutional Networks
[2] Klicpera et al.: Predict then Propagate: Graph Neural Networks meet Personalized PageRank
[3] Xu et al.: Representation Learning on Graphs with Jumping Knowledge Networks
[4] Hamilton et al.: Inductive Representation Learning on Large Graphs
[5] Kipf and Welling: Semi-Supervised Classification with Graph Convolutional Networks
[6] Shchur et al.: Pitfalls of Graph Neural Network Evaluation
----------------------------
Update after the rebuttal: The authors have addressed several issues and improved their manuscript. I greatly appreciate the effort and the new experimental results. However, the main weak point that the novelty of the approach is limited remains valid of course. Therefore, I am still more inclined to rejecting the paper. I have raised my score from "1: Reject" to "3: Weak Reject".
I have raised my score from "5: Weak Reject" to "6: Weak Accept".