TOWARDS BETTER UNDERSTANDING OF GRADIENT-BASED ATTRIBUTION METHODS FOR DEEP NEURAL NETWORKS
Understanding the flow of information in Deep Neural Networks (DNNs) is a challenging problem that has gain increasing attention over the last few years. While several methods have been proposed to explain network predictions, there have been only a few attempts to compare them from a theoretical perspective. What is more, no exhaustive empirical comparison has been performed in the past. In this work, we analyze four gradient-based attribution methods and formally prove conditions of equivalence and approximation between them. By reformulating two of these methods, we construct a unified framework which enables a direct comparison, as well as an easier implementation. Finally, we propose a novel evaluation metric, called Sensitivity-n and test the gradient-based attribution methods alongside with a simple perturbation-based attribution method on several datasets in the domains of image and text classification, using various network architectures.

The paper summarizes and compares some of the current explanation techniques for deep neural networks that rely on the redistribution of relevance / contribution values from the output to the input space.
The main contributions are the introduction of a unified framework that expresses 4 common attribution techniques (Gradient * Input, Integrated Gradient, eps-LRP and DeepLIFT) in a similar way as modified gradient functions and the definition of a new evaluation measure ('sensitivity n') that generalizes the earlier defined properties of 'completeness' and 'summation to delta'.
The unified framework is very helpful since it points out equivalences between the methods and makes the implementation of eps-LRP and DeepLIFT substantially more easy on modern frameworks. However, as correctly stated by the authors some of the unification (e.g. relation between LRP and Gradient*Input) has been already mentioned in prior work.
Sensitivity-n as a measure tries to tackle the difficulty of estimating the importance of features that can be seen either separately or in combination. While the measure shows interesting trends towards a linear behaviour for simpler methods, it does not persuade me as a measure of how well the relevance attribution method mimics the decision making process and does not really point out substantial differences between the different methods. Furthermore, The authors could comment on the relation between sensitivity-n and region perturbation techniques (Samek et al., IEEE TNNLS, 2017). Sensitivtiy-n seems to be an extension of the region perturbation idea to me.
It would be interesting to see the relation between the "unified" gradient-based explanation methods and approaches (e.g. Saliency maps, alpha-beta LRP, Deep Taylor, Deconvolution Networks, Grad-CAM, Guided Backprop ...) which do not fit into the unification framework. It's good that the author mention these works, still it would be great to see more discussion on the advantages/disadvantages, because these methods may have some nice theoretically properties (see e.g. the discussion on gradient vs. decompositiion techniques in Montavon et al., Digital Signal Processing, 2017) which can not be incorporated into the unified framework.


In this paper, a deep boosting algorithm is developed to learn more discriminative ensemble classifier by seamlessly combining a set of base deep CNNs (base experts) with diverse capabilities, e.g., these base deep CNNs are sequentially trained to recognize a set of object classes in an easy-to-hard way according to their learning complexities. Our experimental results have demonstrated that our deep boosting algorithm can significantly improve the accuracy rates on largescale visual recognition.

This paper consider a version of boosting where in each iteration only class weights are updated rather than sample weights and apply that to a series of CNNs for object recognition tasks.
While the paper is comprehensive in their derivations (very similar to original boosting papers and in many cases one to one translation of derivations), it lacks addressing a few fundamental questions:
- AdaBoost optimises exponential loss function via functional gradient descent in the space of weak learners. It's not clear what kind of loss function is really being optimised here. It feels like it should be the same, but the tweaks applied to fix weights across all samples for a class doesn't make it not clear what is that really gets optimised at the end.
- While the motivation is that classes have different complexities to learn and hence you might want each base model to focus on different classes, it is not clear why this methods should be better than normal boosting: if a class is more difficult, it's expected that their samples will have higher weights and hence the next base model will focus more on them. And crudely speaking, you can think of a class weight to be the expectation of its sample weights and you will end up in a similar setup.
- Choice of using large CNNs as base models for boosting isn't appealing in practical terms, such models will give you the ability to have only a few iterations and hence you can't achieve any convergence that often is the target of boosting models with many base learners.
- Experimentally, paper would benefit with better comparisons and studies: 1) state-of-the-art methods haven't been compared against (e.g. ImageNet experiment compares to 2 years old method) 2) comparisons to using normal AdaBoost on more complex methods haven't been studied (other than the MNIST) 3) comparison to simply ensembling with random initialisations.
Other comments:
- Paper would benefit from writing improvements to make it read better.
- "simply use the weighted error function": I don't think this is correct, AdaBoost loss function is an exponential loss. When you train the base learners, their loss functions will become weighted.
-  "to replace the softmax error function (used in deep learning)": I don't think we have softmax error function


Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space. While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs. We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data. We find that adversarial training increases the robustness of classifiers against shared perturbations. Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations. Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs. However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or Gaussian blurring. It thus makes successful attacks on the classifier in the physical world less likely. Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs.

Summary:
This paper empirically studies adversarial perturbations dx and what the effects are of adversarial training (AT) with respect to shared (dx fools for many x) and singular (only for a single x) perturbations. Experiments use a (previously published) iterative fast-gradient-sign-method and use a Resnet on CIFAR.
The authors conclude that in this experimental setting:
- AT seems to defend models against shared dx's.
- This is visible on universal perturbations, which become less effective as more AT is applied.
- AT decreases the effectiveness of adversarial perturbations, e.g. AT decreases the number of adversarial perturbations that fool both an input x and x with e.g. a contrast change.
- Singular perturbations are easily detected by a detector model, as such perturbations don't change much when applying AT.
Pro:
- Paper addresses an important problem: qualitative / quantitative understanding of the behavior of adversarial perturbations is still lacking.
- The visualizations of universal perturbations as they change during AT are nice.
- The basic observation wrt the behavior of AT is clearly communicated.
Con:
- The experiments performed are interesting directions, although unfocused and rather limited in scope. For instance, does the same phenomenon happen for different datasets? Different models?
- What happens when we use adversarial attacks different from FGSM? Do we get similar results?
- The papers lacks a more in-depth theoretical analysis. Is there a principled reason AT+FGSM defends against universal perturbations?
Overall:
- As is, it seems to me the paper lacks a significant central message (due to limited and unfocused experiments) or significant new theoretical insight into the effect of AT. A number of questions addressed are interesting starting points towards a deeper understanding of *how* the observations can be explained and more rigorous empirical investigations.
Detailed:
-


Policy gradients methods often achieve better performance when the change in policy is limited to a small Kullback-Leibler divergence. We derive policy gradients where the change in policy is limited to a small Wasserstein distance (or trust region). This is done in the discrete and continuous multi-armed bandit settings with entropy regularisation. We show that in the small steps limit with respect to the Wasserstein distance W 2 , policy dynamics are governed by the heat equation, following the Jordan-Kinderlehrer-Otto result. This means that policies undergo diffusion and advection, concentrating near actions with high reward. This helps elucidate the nature of convergence in the probability matching setup, and provides justification for empirical practices such as Gaussian policy priors and additive gradient noise.

The paper ‘Diffusing policies: Towards Wasserstein policy gradient flows’ explores
the connections between reinforcement learning and the theory of quadratic optimal transport (i.e.
using the Wasserstein_2 as a regularizer of an iterative problem that converges toward
an optimal policy). Following a classical result from Jordan-Kinderlehrer-Otto, they show that
the policy dynamics are governed by the heat equation, that translates in an advection-diffusion
scheme. This allows to draw insights on the convergence of empirical practices in the field.
The paper is clear and well-written, and provides a comprehensive survey of known results in the
field of Optimal Transport. The insights on why empirical strategies such as additive gradient noise
are very interesting and helps in understanding why they work in practical settings. That being said,
most of the results presented in the paper are already known (e.g. from the book of Samtambrogio or the work
of G. Peyré on entropic Wasserstein gradient flows) and it is not exactly clear what are the original
contributions of the paper. The fact that the objective is to learn policies
has little to no impact on the derivations of calculus. It clearly suggests that the entropy
regularized Wasserstein_2 distance should be used in numerical experiments but this point is not
supported by experimental results. Their direct applications is rapidly ruled out by highlighting the
computational complexity of solving such gradient flows but in the light of recent papers (see
the work of Genevay https://arxiv.org/abs/1706.00292 or another paper submitted to ICLR on large scale optimal transport
https://openreview.net/forum?id=B1zlp1bRW) numerical applications should be tractable. For these reasons
I feel that the paper would clearly be more interesting for the practitioners (and maybe to some extent
for the audience of ICLR) if numerical applications of the presented theory were discussed or sketched
in classical reinforcement learning settings.
Minor comments:
 - in Equation (10) why is there a ‘d’ in front of the coupling \gamma ?
 - in Section 4.5, please provide references for why numerical estimators of gradient of Wasserstein distances
are biased.


Published as a conference paper at ICLR 2018 UNSUPERVISED NEURAL MACHINE TRANSLATION
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French → English and German → English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project 1 .

unsupervised neural machine translation
This is an interesting paper on unsupervised MT. It trains a standard architecture using:
1) word embeddings in a shared embedding space, learned using a recent approach that works with only tens of bilingual word papers.
2) A encoder-decoder trained using only monolingual data (should cite http://www.statmt.org/wmt17/pdf/WMT15.pdf). Training uses a “denoising” method which is not new: it uses the same idea as contrastive estimation (http://www.aclweb.org/anthology/P05-1044, a well-known method which should be cited).
3) Backtranslation.
All though none of these ideas are new, they haven’t been combined in this way before, and that what’s novel here. The paper is essentially a neat application of (1), and is an empirical/ systems paper. It’s essentially a proof-of-concept that it is that it’s possible to get anything at all using no parallel data. That’s surprising and interesting, but I learned very little else from it. The paper reads as preliminary and rushed, and I had difficulty answering some basic questions:
* In Table (1), I’m slightly puzzled by why 5 is better than 6, and this may be because I’m confused about what 6 represents. It would be natural to compare 5 with a system trained on 100K parallel text, since the systems would then (effectively) differ only in that 5 also exploits additional monolingual data. But the text suggests that 6 is trained on much more than 100K parallel sentences; that is, it differs in at least two conditions (amount of parallel text and use of monolingual text). Since this paper’s primary contribution is empirical, this comparison should be done in a carefully controlled way, differing each of these elements in turn.
* I’m very confused by the comment on p. 8 that “the modifications introduced by our proposal are also limiting” to the “comparable supervised NMT system”. According to the paper, the architecture of the system is unchanged, so why would this be the case? This comment makes it seem like something else has been changed in the baseline, which in turn makes it somewhat hard to accept the results here.
Comment:
* The qualitative analysis is not really an analysis: it’s just a few cherry-picked examples and some vague observations. While it is useful to see that the system does indeed generate nontrivial content in these cases, this doesn’t give us further insight into what the system does well or poorly outside these examples. The BLEU scores suggest that it also produces many low-quality translations. What is different about these particular examples? (Aside: since the cross-lingual embedding method is trained on numerals, should we be concerned that the system fails at translating numerals?)
Questions:
* Contrastive estimation considers other neighborhood functions (“random noise” in the parlance of this paper), and it’s natural to wonder what would happen if this paper also used these or other neighborhood functions. More importantly, I suspect the the neighborhood functions are important: when translating between Indo-European languages as in these experiments, local swaps are reasonable; but in translating between two different language families (as would often be the case in the motivating low-resource scenario that the paper does not actually test), it seems likely that other neighborhood functions would be important, since structural differences would be much larger.
Presentational comments (these don’t affect my evaluation, they’re mostly observations but they contribute to a general feeling that the paper is rushed and preliminary):
* BPE does not “learn”, it’s entirely deterministic.
* This paper is at best tangentially related to decipherment. Decipherment operates under two quite different assumptions: there is no training data for the source language ciphertext, only the ciphertext itself (which is often very small); and the replacement function is deterministic rather than probabilistic (and often monotonic). The Dou and Knight papers are interesting, but they’re an adaptation of ideas rather than decipherment per se. Since none of those ideas are used here this feels like hand-waving.
* Future work is vague: “we would like to detect and mitigate the specific causes…” “we also think that a better handling of rare words…” That’s great, but how will you do these things? Do you have specific reasons to think this, or ideas on how to approach them? Otherwise this is just hand-waving.


Published as a conference paper at ICLR 2018 DEEP AUTOENCODING GAUSSIAN MIXTURE MODEL FOR UNSUPERVISED ANOMALY DETECTION
Unsupervised anomaly detection on multi-or high-dimensional data is of great importance in both fundamental machine learning research and industrial applications, for which density estimation lies at the core. Although previous approaches based on dimensionality reduction followed by density estimation have made fruitful progress, they mainly suffer from decoupled model learning with inconsistent optimization goals and incapability of preserving essential information in the low-dimensional space. In this paper, we present a Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection. Our model utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input data point, which is further fed into a Gaussian Mixture Model (GMM). Instead of using decoupled two-stage training and the standard Expectation-Maximization (EM) algorithm, DAGMM jointly optimizes the parameters of the deep autoencoder and the mixture model simultaneously in an end-to-end fashion, leveraging a separate estimation network to facilitate the parameter learning of the mixture model. The joint optimization, which well balances autoencoding reconstruction, density estimation of latent representation, and regularization, helps the autoencoder escape from less attractive local optima and further reduce reconstruction errors, avoiding the need of pre-training. Experimental results on several public benchmark datasets show that, DAGMM significantly outperforms state-of-the-art anomaly detection techniques, and achieves up to 14% improvement based on the standard F 1 score.

The paper presents a new technique for anomaly detection where the dimension reduction and the density estimation steps are jointly optimized. The paper is rigorous and ideas are clearly stated. The idea to constraint the dimension reduction to fit a certain model, here a GMM, is relevant, and the paper provides a thorough comparison with recent state-of-the-art methods. My main concern is that the method is called unsupervised, but it uses the class information in the training, and also evaluation. I'm also not convinced of how well the Gaussian model fits the low-dimensional representation and how well can a neural network compute the GMM mixture memberships.
1. The framework uses the class information, i.e., “only data samples from the normal class are used for training”, but it is still considered unsupervised. Also, the anomaly detection in the evaluation step is based on a threshold which depends on the percentage of known anomalies, i.e., a priori information. I would like to see a plot of the sample energy as a function of the number of data points. Is there an elbow that indicates the threshold cut? Better yet it would be to use methods like Local Outlier Factor (LOF) (Breunig et al., 2000 – LOF:Identifying Density-based local outliers) to detect the outliers (these methods also have parameters to tune, sure, but using the known percentage of anomalies to find the threshold is not relevant in a purely unsupervised context when we don't know how many anomalies are in the data).
2. Is there a theoretical justification for computing the mixture memberships for the GMM using a neural network?
3. How do the regularization parameters \lambda_1 and \lambda_2 influence the results?
4. The idea to jointly optimize the dimension reduction and the clustering steps was used before neural nets (e.g., Yang et al., 2014 -  Unsupervised dimensionality reduction for Gaussian mixture model). Those approaches should at least be discussed in the related work, if not compared against.
5. The authors state that estimating the mixture memberships with a neural network for GMM in the estimation network instead of the standard EM algorithm works better. Could you provide a comparison with EM?
6. In the newly constructed space that consists of both the extracted features and the representation error, is a Gaussian model truly relevant? Does it well describe the new space? Do you normalize the features (the output of the dimension reduction and the representation error are quite different)? Fig. 3a doesn't seem to show that the output is a clear mixture of Gaussians.
7. The setup of the KDDCup seems a little bit weird, where the normal samples and anomalies are reversed (because of percentage), where the model is trained only on anomalies, and it detects normal samples as anomalies ... I'm not convinced that it is the best example, especially that is it the one having significantly better results, i.e. scores ~ 0.9 vs. scores ~0.4/0.5 score for the other datasets.
8. The authors mention that “we can clearly see from Fig. 3a that DAGMM is able to well separate ...” - it is not clear to me, it does look better than the other ones, but not clear. If there is a clear separation from a different view, show that one instead. We don't need the same view for all methods.
9. In the experiments the reduced dimension used is equal to 1 for two of the experiments and 2 for one of them. This seems very drastic!
Minor comments:
1. Fig.1: what dimension reduction did you use? Add axis labels.
2. “DAGMM preserves the key information of an input sample” - what does key information mean?
3. In Fig. 3 when plotting the results for KDDCup, I would have liked to see results for the best 4 methods from Table 1, OC-SVM performs better than PAE. Also DSEBM-e and DSEBM-r seems to perform very well when looking at the three measures combined. They are the best in terms of precision.
4. Is the error in Table 2 averaged over multiple runs? If yes, how many?
Quality – The paper is thoroughly written, and the ideas are clearly presented. It can be further improved as mentioned in the comments.
Clarity – The paper is very well written with clear statements, a pleasure to read.
Originality – Fairly original, but it still needs some work to justify it better.
Significance – Constraining the dimension reduction to fit a certain model is a relevant topic, but I'm not convinced of how well the Gaussian model fits the low-dimensional representation and how well can a neural network compute the GMM mixture memberships.


Published as a conference paper at ICLR 2018 BOUNDARY-SEEKING GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs, Goodfellow et al., 2014) are a learning framework that rely on training a discriminator to estimate a measure of difference between a target and generated distributions. GANs, as normally formulated, rely on the generated samples being completely differentiable w.r.t. the generative parameters, and thus do not work for discrete data. We introduce a method for training GANs with discrete data that uses the estimated difference measure from the discriminator to compute importance weights for generated samples, thus providing a policy gradient for training the generator. The importance weights have a strong connection to the decision boundary of the discriminator, and we call our method boundary-seeking GANs (BGANs). We demonstrate the effectiveness of the proposed algorithm with discrete image and character-based natural language generation. In addition, the boundary-seeking objective extends to continuous data, which can be used to improve stability of training, and we demonstrate this on Celeba, Large-scale Scene Understanding (LSUN) bedrooms, and Imagenet without conditioning.

Thanks for the feedback and for clarifying the 1) algorithm and the assumptions in the multivariate case 2) comparison to RL based methods 3) connection to estimating importance sampling weights using GAN discriminator.
I think the paper contribution is now more clear and strengthened with additional convincing experiments and I am increasing my score to 7.
The paper would still benefit from doing the experiment with importance weights by pixel , rather then a global one as done in the paper now. I encourage the authors to still do the experiment, see if there is any benefit.
==== Original Review =====
Summary of the paper:
The paper presents a method based on importance sampling and reinforcement learning to learn discrete generators in the GAN framework. The GAN uses an  f-divergence cost function for  training the discriminator. The generator is trained to minimize the KL distance between the  discrete generator q_{\theta}(x|z), and the importance weight discrete real distribution estimator w(x|z)q(\theta|z). where w(x|z) is estimated in turn using the discriminator.
The methodology is also extended to the continuous case. Experiments are conducted on quantized image generation, and text generation.
Quality:
the paper is overall well written and supported with reasonable experiments.
Clarity:
The paper has a lot of typos that make sometimes the paper harder to follow:
- page (2) Eq 3 max , min should be min, max if we want to keep working with f-divergence
- Definition 2.1 \mathbb{Q}_{\theta} --> \mathbb{Q}
- page 5 the definition of \tilde{w}(x^{(m})) in the normalization it is missing \tilde{w}
- Equation (10) \nabla_{\theta}\log(x|z) --> \nabla_{\theta}\log(x^{(m)}|z)
- In algorithm 1, again missing indices in the update of theta  --> \nabla_{\theta}\log(x^{(m|n)}|z^{n})

Originality:
The main ingredients of the paper are well known and already used in the literature (Reinforce for discrete GAN with Disc as a reward for e.g GAN for image captioning Dai et al). The perspective from importance sampling coming from f-divergence for discrete GAN has some novelty although the foundations of this work relate also to previous work:
- Estimating ratios using the discriminator is well known for e.g learning implicit models , Mohamed et al
- The relation of  importance sampling to  reinforce is also well known" On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient," Tang and Abbeel.
General Review:
- when the generator is producing only *one* discrete distribution the theory is presented in Section 2.3. When we move to experiments, for image generation for example, we need to have a generator that produces a distribution by pixel. It would be important for 1) understanding the work 2) the reproducibility of the work to parallel algorithm 1 and have it *in the paper*, for this 'multi discrete distribution ' generation case.  If we have N pixels    \log(p(x_1,...x_N|z))= \Pi_i g_{\theta}(x_i|z) (this should be mentioned in the paper if it is the case ), it would be instructive to comment on the assumptions on independence/conditional dependence of this model, also to state clearly how the generator is updated in this case and what are importance sampling weights.
- Would it make sense in this N pixel discrete case generation to have also the discriminator produce N probabilities of real and fake as in PixelGAN in Isola et al? then see in this case what are the importance sampling weights this would parallel the instantaneous reward in RL?


GUIDE ACTOR-CRITIC FOR CONTINUOUS CONTROL
Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this paper, we propose a novel actor-critic method called the guide actor-critic (GAC). GAC firstly learns a guide actor that locally maximizes the critic and then it updates the policy parameter based on the guide actor by supervised learning. Our main theoretical contributions are two folds. First, we show that GAC updates the guide actor by performing second-order optimization in the action space where the curvature matrix is based on the Hessians of the critic. Second, we show that the deterministic policy gradient method is a special case of GAC when the Hessians are ignored. Through experiments, we show that our method is a promising reinforcement learning method for continuous controls.

The paper presents a clever trick for updating the actor in an actor-critic setting: computing a guide actor that diverges from the actor to improve critic value, then updating the actor parameters towards the guide actor. This can be done since, when the parametrized actor is Gaussian and the critic value can be well-approximated as quadratic in the action, the guide actor can be optimized in closed form.
The paper is mostly clear and well-presented, except for two issues: 1) there is virtually nothing novel presented in the first half of the paper (before Section 3.3); and 2) the actual learning step is only presented on page 6, making it hard to understand the motivation behind the guide actor until very late through the paper.
The presented method itself seems to be an important contribution, even if the results are not overwhelmingly positive. It'd be interesting to see a more elaborate analysis of why it works well in some domains but not in others. More trials are also needed to alleviate any suspicion of lucky trials.
There are some other issues with the presentation of the method, but these don't affect the merit of the method:
1. Returns are defined from an initial distribution that is stationary for the policy. While this makes sense in well-mixing domains, the experiment domains are not well-mixing for most policies during training, for example a fallen humanoid will not get up on its own, and must be reset.
2. The definition of beta(a|s) as a mixture of past actors is inconsistent with the sampling method, which seems to be a mixture of past trajectories.
3. In the first paragraph of Section 3.3: "[...] the quality of a guide actor mostly depends on the accuracy of Taylor's approximation." What else does it depend on? Then: "[...] the action a_0 should be in a local vicinity of a."; and "[...] the action a_0 should be similar to actions sampled from pi_theta(a|s)." What do you mean "should"? In order for the Taylor approximation to be good?
4. The line before (19) is confusing, since (19) is exact and not an approximation. For the approximation (20), it isn't clear if this is a good approximation. Why/when is the 2nd term in (19) small?
5. The parametrization nu of \hat{Q} is never specified in Section 3.6. This is important in order to evaluate the complexities involved in computing its Hessian.


While domain adaptation has been actively researched in recent years, most theoretical results and algorithms focus on the single-source-single-target adaptation setting. Naive application of such algorithms on multiple source domain adaptation problem may lead to suboptimal solutions. We propose a new generalization bound for domain adaptation when there are multiple source domains with labeled instances and one target domain with unlabeled instances. Compared with existing bounds, the new bound does not require expert knowledge about the target distribution, nor the optimal combination rule for multisource domains. Interestingly, our theory also leads to an efficient learning strategy using adversarial neural networks: we show how to interpret it as learning feature representations that are invariant to the multiple domain shifts while still being discriminative for the learning task. To this end, we propose two models, both of which we call multisource domain adversarial networks (MDANs): the first model optimizes directly our bound, while the second model is a smoothed approximation of the first one, leading to a more data-efficient and task-adaptive model. The optimization tasks of both models are minimax saddle point problems that can be optimized by adversarial training. To demonstrate the effectiveness of MDANs, we conduct extensive experiments showing superior adaptation performance on three real-world datasets: sentiment analysis, digit classification, and vehicle counting.

Quality:
The paper appears to be correct.
Clarity:
The paper is very clear
Originality:
The theoretical contribution extends the seminal work of Ben-David et al., the idea of using adversarial learning is not new, the novelty is mediaum
Significance:
The theoretical analysis is interested but for me limited, the idea of the algorithm is not new but as far as I know the first explicitly presented for multi-source.
Pros:
-new theoretical analysis for multisource problem
-paper clear
-smoothed version is interesting
Cons
-Learning bounds with worst case standpoint is probably not the best analysis for multisource learning
-experimental evaluation limited in the sense that similar algorithms in the literature are not compared
-Extension a bit direct from the seminal work of Ben-David et al.
Summary:
This paper presents a multiple source domain adaptation approach based on adversarial learning.
The setting considered contains multiple source domains with labeled instances and one target domain with unlabeled instances. The authors propose learning bounds in this context that extend seminal work of Ben-David and co-authors(2007,2010) where they essentially consider the max source error and the max divergence between target and source with the presence of empirical estimate.
Then, they propose an adversarial algorithm to optimize this bound, with another version optimizing a smoothed version, following the approach of Ganin et al.(2016).
An experimental evaluation on 3 known tasks is presented.
Comments:
Comments:
-I am not particularly convinced that the proposed theory explains best multi-source learning. In multi-source, you expect that one source may compensate the others when needed for classification of particular instances. The paper considers a kind of worst case by taking the max error over the sources and the max divergence between target and source, but not really representative of what happens for real problems in the sense that you do not take into account how the different sources interact.
The experimental results confirm this aspect actually.
Maybe, the authors could propose a learning bound that correspond to the smoothed version proposed in the paper and that works best.
The Hard version of the algorithm seems here to comply with the bound, while the algorithm that is really interesting is the smoothed version.
-Experimental evaluation is a bit limited, there is no comparison with other (deep learning methods) tackling multi-source scenarios (or equivalent), while I think it is easy to find related approaches :
-E. Tzeng, J. Hoffman, T. Darrell, K. Saenko. Simultaneous Deep Transfer Across Domains and Tasks. ICCV 2015.
-I-H Jhuo, D Liu, D.T. Lee, and S-Fu. Chang. Robust visual domain adaptation with low-rank reconstruction. In IEEE CVPR, 2012.
-Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In IEEE International Conference on Computer Vision (ICCV), 2015.
-Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain generalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-R. Gopalan,R. Li,and R. Chellappa. Unsupervised Adaptation Across Domain shifts by generating intermediate data representations. PAMI, 36(11), 2014.
Note also this paper at CVPR'17: about Domain adversarial adaptation.
E. Tzeng, J. Hoffman, K. Saenko, T. Darrell.  Adversarial Discriminative Domain Adaptation, CVPR 2017.
-Nothing is said about the complexity of applying the algorithm on the different datasets (convergence, tuning, ...)
For the smoothed version, it could be interesting to see if the weights w_i associated to each source are related to each (original) source error and see how the sources are complementary.
--
After rebuttal
--
The new results and experimental evaluation have improved the paper. I increased my score.


MINIMAL-ENTROPY CORRELATION ALIGNMENT FOR UNSUPERVISED DEEP DOMAIN ADAPTATION
In this work, we face the problem of unsupervised domain adaptation with a novel deep learning approach which leverages our finding that entropy minimization is induced by the optimal alignment of second order statistics between source and target domains. We formally demonstrate this hypothesis and, aiming at achieving an optimal alignment in practical cases, we adopt a more principled strategy which, differently from the current Euclidean approaches, deploys alignment along geodesics. Our pipeline can be implemented by adding to the standard classification loss (on the labeled source domain), a source-to-target regularizer that is weighted in an unsupervised and data-driven fashion. We provide extensive experiments to assess the superiority of our framework on standard domain and modality adaptation benchmarks.

This paper improves the correlation alignment approach to domain adaptation from two aspects. One is to replace the Euclidean distance by the geodesic Log-Euclidean distance between two covariance matrices. The other is to automatically select the balancing cost by the entropy on the target domain. Experiments are conducted from SVHN to MNIST and from SYN MNIST to SVHN. Additional experiments on cross-modality recognition are reported from RGB to depth.
Strengths:
+ It is a sensible idea to improve the Euclidean distance by the geodesic Log-Euclidean distance to better explore the manifold structure of the PSD matrices.
+ It is also interesting to choose the balancing cost using the entropy on the target. However, this point is worth further exploring (please see below for more detailed comments).
+ The experiments show that the geodesic correlation alignment outperforms the original alignment method.
Weaknesses:
- It is certainly interesting to have a scheme to automatically choose the hyper-parameters in unsupervised domain adaptation, and the entropy over the target seems like a reasonable choice. This point is worth further exploring for the following reasons.
1. The theoretical result is not convincing given it relies on many unrealistic assumptions, such as the null performance degradation under perfect correlation alignment, the Dirac’s delta function as the predictions over the target, etc.
2. The theorem actually does not favor the correlation alignment over the geodesic alignment. It does not explain that, in Figure 2, the entropy is able to find the best balancing cost \lamba for geodesic alignment but not for the Euclidean alignment.
3. The entropy alignment seems an interesting criterion to explore in general. Could it be used to find fairly good hyper-parameters for the other methods? Could it be used to determine the other hyper-parameters (e..g, learning rate, early stopping) for the geodesic alignment?
4. If one leaves a subset of the target domain out and use its labels for validation, how different would the selected balancing cost \lambda differ from that by the entropy?
- The cross-modality setup (from RGB to depth) is often not considered as domain adaptation. It would be better to replace it by another benchmark dataset. The Office-31 dataset is still a good benchmark to compare different methods and for the study in Section 5.1, though it is not necessary to reach state-of-the-art results on this dataset because, as the authors noted, it is almost saturated.
Question:
- I am not sure how the gradients were computed after the eigendecomposition in equation (8).
I like the idea of automatically choosing free parameters using the entropy over the target domain. However, instead of justifying this point by the theorem that relies on many assumptions, it is better to further test it using experiments (e.g., on Office31 and for other adaptation methods). The geodesic correlation alignment is a reasonable improvement over the Euclidean alignment.


Planning problems are among the most important and well-studied problems in artificial intelligence. They are most typically solved by tree search algorithms that simulate ahead into the future, evaluate future states, and back-up those evaluations to the root of a search tree. Among these algorithms, Monte-Carlo tree search (MCTS) is one of the most general, powerful and widely used. A typical implementation of MCTS uses cleverly designed rules, optimised to the particular characteristics of the domain. These rules control where the simulation traverses, what to evaluate in the states that are reached, and how to back-up those evaluations. In this paper we instead learn where, what and how to search. Our architecture, which we call an MCTSnet, incorporates simulation-based search inside a neural network, by expanding, evaluating and backing-up a vector embedding. The parameters of the network are trained end-to-end using gradient-based optimisation. When applied to small searches in the well-known planning problem Sokoban, the learned search algorithm significantly outperformed MCTS baselines.

This paper designs a deep learning architecture that mimics the structure of the well-known MCTS algorithm. From gold standard state-action pairs, it learns each component of this architecture in order to predict similar actions.
I enjoyed reading this paper. The presentation is very clear, the design of the architecture is beautiful, and I was especially impressed with the related work discussion that went back to identify other game search and RL work that attempts to learn parts of the search algorithm. Nice job overall.
The main flaw of the paper is in its experiments. If I understand them correctly, the comparison is between a neural network that has been learned on 250,000 trajectories of 60 steps each where each step is decided by a ground truth close-to-optimal algorithm, say MCTS with 1000 rollouts (is this mentioned in the paper). That makes for a staggering 15 billion rollouts of prior data that goes into the MCTSNet model. This is compared to 25 rollouts of MCTS that make the decision for the baseline. I suspect that generating the training data and learning the model takes an enormous amount of CPU time, while 25 MCTS rollouts can probably be done in a second or two. I'm sure I'm misinterpreting some detail here, but how is this a fair comparison?
Would it be fair to have a baseline that learns the MCTS coefficient on the training data? Or one that uses the value function that was learned with classic search? I find it difficult to understand the details of the experimental setup, and maybe some of these experiments are reported. Please clarify. Also: colors are not distinguishable in grey print.
How would the technique scale with more MCTS iterations? I suspect that the O(N^2) complexity is very prohibitive and will not allow this to scale up?
I'm a bit worried about the idea of learning to trade off exploration and exploitation. In the end you'll just allow for the minimal amount of exploration to solve the games you've already seen. This seems risky, and I suspect that UCB and more statistically principled approaches would be more robust in this regard?
Are these Sokoban puzzles easy for classical AI techniques? I know that many of them can be solved by A* search with a decent heuristic. It would be fair to discuss this.
The last sentence of conclusions is too far reaching; there is really no evidence for that claim.


Published as a conference paper at ICLR 2018 MATRIX CAPSULES WITH EM ROUTING
A capsule is a group of neurons whose outputs represent different properties of the same entity. Each layer in a capsule network contains many capsules. We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose). A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships. Each of these votes is weighted by an assignment coefficient. These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes. The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers. On the smallNORB benchmark, capsules reduce the number of test errors by 45% compared to the state-of-the-art. Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network.

The paper proposes a novel architecture for capsule networks. Each capsule has a logistic unit representing the presence of an entity plus a 4x4 pose matrix representing the entity/viewer relationship. This new representation comes with a novel iterative routing scheme, based on the EM algorithm.
Evaluated on the SmallNORB dataset, the approach proves to be more accurate than previous work (beating also the recently proposed "routing-by-agreement" approach for capsule networks by Sabour et al.). It also generalizes well to new, unseen viewpoints and proves to be more robust to adversarial examples than traditional CNNs.
Capsule networks have recently gained attention from the community. The paper addresses important shortcomings exhibited by previous work (Sabour et al.), introducing a series of valuable technical novelties.
There are, however, some weaknesses. The proposed routing scheme is quite complex (involving an EM-based step at each layer); it's not fully clear how efficiently it can be performed / how scalable it is. Evaluation is performed on a small dataset for shape recognition; as noted in Sec. 6, the approach will need to be tested on larger, more challenging datasets. Clarity could be improved in some parts of the paper (e.g.: Sec. 1.1 may not be fully clear if the reader is not already familiar with (Sabour et al., 2017); the authors could give a better intuition about what is kept and what is discarded, and why, from that approach. Sec. 2: the sentence "this is incorrect because the transformation matrix..." could be elaborated more. V_{ih} in eq. 1 is defined only a few lines below; perhaps, defining the variables before the equations could improve clarity. Sec. 2.1 could be accompanied by mathematical formulation).
All in all, the paper brings an original contribution and will encourage further research / discussion on an important research question (how to effectively leverage knowledge about the part-whole relationships).
Other notes:
- There are a few typos (e.g. Sec. 1.2 "(Jaderberg et al. (2015)",  Sec. 2 "the the transformation", Sec. 4 "cetral crop" etc.).
- The authors could discuss in more detail why the approach does not show significant improvement on NORB with respect to the state of the art.
- The authors could provide more insights about why capsule gradients are smaller than CNN ones.
- It would be interesting to discuss how the network could potentially be adapted, in the future, to: 1. be more efficient 2. take into account other changes produced by viewpoint changes (pixel intensities, as noted in Sec. 1).
- In Sec, 4, the authors could provide more details about the network training.
- In Procedure 1, for indexing tensors and matrices it might be better to use a comma to separate dimensions (e.g. V_{:,c,:} instead of V_{:c:}).


This paper develops a novel methodology for using symbolic knowledge in deep learning. From first principles, we derive a semantic loss function that bridges between neural output vectors and logical constraints. This loss function captures how close the neural network is to satisfying the constraints on its output. An experimental evaluation shows that our semantic loss function effectively guides the learner to achieve (near-)state-of-the-art results on semi-supervised multi-class classification. Moreover, it significantly increases the ability of the neural network to predict structured objects, such as rankings and paths. These discrete concepts are tremendously difficult to learn, and benefit from a tight integration of deep learning and symbolic reasoning methods.

SUMMARY
The paper proposes a new form of regularization utilizing logical constraints. The semantic loss function is built on the exploitation of symbolic knowledge extracted from data and connecting the logical constraints to the outputs of a neural network. The use of Boolean logic as a constraint provides a secondary regularization term to prevent over-fitting and improve predictions. The benefit of using the function is found primarily with semi-supervised tasks where data is partially unlabelled. The logical constraints provided by the semantic loss function allow for improved classification of unlabeled data.
Output constraints for the semantic loss function are represented with one-hot encoding, prefer- ence rankings, and paths in a grid. These three different output constraints are designed to explore different learning purposes. The semantic function was tested on both semi-supervised classifica- tion tasks as well as structure learning. The paper primarily focuses on the one-hot encoding constraint as it is viewed as a capable technique for multi-class classification.
POSITIVES
In terms of structure, the paper was written very well. Sufficient background information was con- veyed which helped in understanding the proposed semantic loss function. A thorough breakdown is also carried out on the semantic loss function itself by explaining its axioms which help explain how the outputs of a neural network match a given constraint.
As a scientific contribution, I would say results from the experiments were able to justify the proposal of the semantic loss function. The function was able to perform better than most other implementations for semi-supervised learning tasks, and the function was tested on multiple datasets. The paper also made use of testing the function against other notable machine learning approaches, and in most cases the function performed better, but this usually was confined to semi-supervised learning tasks. During supervised learning tasks the function did not perform markedly better than older implementations. Given that, the semantic loss function did prove to be a seemingly simple approach to improving semi-supervised classification tasks.
• The background section covers the knowledge required in understanding the semantic loss function. The paper also clearly explains the meaning for some of the notation used in the definitions.
• Experiments which clearly show the benefit of using the semantic loss function. Multiple experiment types were done as well which showed evidence of the broad applicability of the function.
• In depth description of the definitions, axioms, and propositions of the semantic loss function.
• A large number of experiments exploring the usefulness of the function for multiple learning tasks, and on multiple datasets.
NEGATIVES
I was not clear if the logical constraints are to be instantiated before learning, i.e. they are defined by hand prior to being implemented in the neural network. This is a pretty important question and drastically changes the nature of the learning process. Beyond that complaint, the paper did not suffer from any critical issues. There were some issues with spelling, and the section titled ’Algorithm’ fails to clearly define a complete algorithm using the semantic loss function. It would have helped to have two algorithms. One defining the pipeline for the semantic loss function, and another showing the implementation of the function in a machine learning framework. The semantic loss function found success only in cases were the learning task was semi-supervised, and not in cases of total supervised learning. This is not a true negative, but an observation on the effectiveness of the function.
- A few typos in the paper.
- The axioms for the semantic loss function where defined but there seemed to be a lack of a clear algorithm provided showing the pipeline implementation of the semantic loss function.
- While the semantic loss function does improve learning performance in most cases, the im- provements are confined to semi-supervised learning tasks, and with the MNIST dataset another methodology, Ladder Nets, was able to outperform the semantic loss function.
RELATED WORK
The paper proposed that logic constraints applied to the output of neural networks have the capacity to improve semi-supervised classification tasks as well as finding the shortest path. In the introduction, the paper lists Zhiting Hu et al. paper titled Harnessing Deep Neural Networks with Logic Rules as an example of a similar approach. Hu et al. paper utilized logic constraints in conjunction with neural nets as well. A key difference was that Hu et al. applied their network architecture to supervised classification tasks. Since the performance of the current papers semantic loss function with supervised tasks did not improve upon other methods, it may benefit to utilize the research by Hu et al. as a means of direct comparison for supervised learning tasks, and possibly incorporate their methods with the semantic loss function in order to improve upon supervised learning tasks.
CONCLUSION
Given the success of the semantic loss function with semi-supervised tasks, I would accept this paper. The semantic loss was able to improve learning with respect to the tested datasets, and the paper clearly described the properties of the functions. The paper would benefit by including a more concrete algorithm describing the flow of data through a given neural net to the semantic loss function, as well as the process by which the semantic loss function constrains the data based on propositional logic, but in general this complaint is more nit picking. The semantic loss function and the experiments which tested the function showed clearly that there is a benefit to this research and there are areas for it to improve.


Workshop track -ICLR 2018 A FLEXIBLE APPROACH TO AUTOMATED RNN ARCHITECTURE GENERATION
The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components. We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width. The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization. Using two different candidate generation techniques, random search with a ranking function and reinforcement learning, we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains. The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.

This paper investigates meta-learning strategy for automated architecture search in the context of RNN. To constraint the architecture search space, authors propose a DSL that specifies the RNN recurrent operations. This DSL allows to explore RNN architectures using either random search or a reinforcement-learning strategy. Candidate architectures are ranked using a TreeLSTM that tries to predict the architecture performances. The top-k architectures are then evaluated by fully training them on a given task.
Authors evaluate their approach on  PTB/Wikitext 2 language modeling and Multi30k/IWSLT'16  machine translation. In both experiments, authors show that their approach obtains competitive results and can sometime outperforms RNN cells such as GRU/LSTM. In the PTB experiment, their architecture however underperforms other LSTM variant in the literatures.
- Quality/Clarity
The paper is overall well written and pleasant to read.
Few details can be clarified. In particular how did you initialize the weight and bias for both the LSTM/GRU baselines and the found architectures? Is there other works leveraging RNN that report results on the Multi30k/IWSLT datasets?
You state in paragraph 3.2 that human experts can inject the previous best known architecture when training the ranking networks. Did you use this in the experiments? If yes, what was the impact of this online learning strategy on the final results?
- Originality
The idea of using DSL + ranking for architecture search seems novel.
- Significance
Automated architecture search is a promising way to design new networks. However, it is not clear why the proposed approach is not able to outperforms other LSTM-based architectures on the PTB task. Could the problem arise from the DSL that constraint too much the search space ? It would be nice to have other tasks that are commonly used as benchmark for RNN to see where this approach stand.
In addition, authors propose both a DSL, a random and RL generator and a ranking function. It would be nice to disentangle the contributions of the different components. In particular, did the authors compare the random search vs the RL based generator or the performances of the RL-based generator when the ranking network is not used?
Although authors do show that they outperform NAScell in one setting, it would be nice to have an extended evaluation (using character level PTB for instance).


We propose a general deep reinforcement learning method and apply it to robot manipulation tasks. Our approach leverages demonstration data to assist a reinforcement learning agent in learning to solve a wide range of tasks, mainly previously unsolved. We train visuomotor policies end-to-end to learn a direct mapping from RGB camera inputs to joint velocities. Our experiments indicate that our reinforcement and imitation approach can solve contact-rich robot manipulation tasks that neither the state-of-the-art reinforcement nor imitation learning method can solve alone. We also illustrate that these policies achieved zero-shot sim2real transfer by training with large visual and dynamics variations.

Paper summary: The authors propose a number of tricks to enable training policies for pick and place style tasks using a combination of GAIL-based imitation learning and hand-specified rewards, as well as use of unobserved state information during training and hand-designed curricula. The results demonstrate manipulation policies for stacking blocks and moving objects, as well as preliminary results for zero-shot transfer from simulation to a real robot for a picking task and an attempt at a stacking task.
Review summary: The paper proposes a limited but interesting contribution that will be especially of interest to practitioners, but the scope of the contribution is somewhat incremental in light of recent work, and the results, while interesting, could certainly be better. In the balance, I think the paper should be accepted, because it will be of value to practitioners, and I appreciate the detail and real-world experiments. However, some of the claims should be revised to better reflect what the paper actually accomplishes: the contribution is a bit limited in places, but that's *OK* -- the authors should just be up-front about it.
Pros:
- Interesting tasks that combine imitation and reinforcement in a logical (but somewhat heuristic) way
- Good simulated results on a variety of pick-and-place style problems
- Some initial attempt at real-world transfer that seems promising, but limited
- Related work is very detailed and I think many will find it to be a very valuable overview
Cons:
- Some of the claims (detailed below) are a bit excessive in my opinion
- The paper would be better if it was scoped more narrowly
- Contribution is a bit incremental and somewhat heuristic
- The experimental results are difficult to interpret in simulation
- The real-world experimental results are not great
- There are a couple of missing citations (but overall related work is great)
Detailed discussion of potential issues and constructive feedback:
> "Our approach leverages demonstration data to assist a reinforcement learning agent in learning to solve a wide range of tasks, mainly previously unsolved."
>> This claim is a bit peculiar. Picking up and placing objects is certainly not "unsolved," there are many examples. If you want image-based pick and place with demonstrations for example, see Chebotar '17 (not cited). If you want stacking blocks, see Nair '17. While it's true that there is a particular combination of factors that doesn't exactly appear in prior work, the statement the authors make is way too strong. Chebotar '17 shows picking and placing a real-world objective with a much higher success rate than reported here, without simulation. Nair '17 shows a much harder stacking task, but without images -- would that method have worked just as well with image-based distillation? Very likely. Rajeswaran '17 shows tasks that arguably are much harder. Maybe a more honest statement is that this paper proposes some tasks that prior methods don't show, and some prior methods show tasks that the proposed method can't solve. But as-is, this statement misrepresents prior work.
> Previous RL-based robot manipulation policies (Nair et al., 2017; Popov et al., 2017) largely rely on low-level states as input, or use severely limited action spaces that ignore the arm and instead learn Cartesian control of a simple gripper. This limits the ability of these methods to represent and solve more complex tasks (e.g., manipulating arbitrary 3D objects) and to deploy in real environments where the privileged state information is unavailable.
>> This is a funny statement. Some use images, some don't. There is a ton of prior work on RL-based robot manipulation that does use images. The current paper does use object state information during training, which some prior works manage to avoid. The comments about Cartesian control are a bit peculiar... the proposed method controls fingers, but the hand is simple. Some prior works have simpler grippers (e.g., Nair) and some have much more complex hands (e.g., Rajeswaran). So this one falls somewhere in the middle. That's fine, but again, this statement overclaims a bit.
> To sidestep the constraints of training on real hardware we embrace the sim2real paradigm which
has recently shown promising results (James et al., 2017; Rusu et al., 2016a).
>> Probably should cite Sadeghi et al. and Tobin et al. in regard to randomization, both of which precede James '17.
> we can, during training, exploit privileged information about the true system state
>> This was done also in Pinto et al. and many of the cited GPS papers
> our policies solve the tasks that the state-of-the-art reinforcement and imitation learning cannot solve
>> I don't think this statement is justified without much wider comparisons -- the authors don't attempt any comparisons to prior work, such as Chebotar '17 (which arguably is closest in terms of demonstrated behaviors), Nair '17 (which is also close but doesn't use images, though it likely could).
> An alternative strategy for dealing with the data demand is to train in simulation and transfer
>> Aside from previously mentioned citations, should probably cite Devin "Towards Adapting Deep Visuomotor Representations"
> Sec 3.2.1
>> This method seems a bit heuristic. It's logical, but can you say anything about what this will converge to? GAIL will try to match the demonstration distribution, and RL will try to maximize expected reward. What will this method do?
> Experiments
>> Would it be possible to indicate some measure of success rate for the simulated experiments? As-is, it's hard to tell how well either the proposed method or the baselines actually work.
> Transfer
>> My reading of the transfer experiments is that they are basically unsuccessful. Picking up a rectangular object with 80% success rate is not very good. The stacking success rate is too low to be useful. I do appreciate the authors trying out their method on a real robotic platform, but perhaps the more honest assessment of the outcome of these experiments is that the approach didn't work very well, and more research is needed. Again, it's *OK* to say this! Part of the purpose of publishing a paper is to stimulate future research directions. I think the transfer experiments should definitely be kept, but the authors should discuss the limitations to help future work address them, and present the transfer appropriately in the intro.
> Diverse Visuomotor Skills
>> I think this is a peculiar thing to put in the title. Is the implication that prior work is not diverse? Arguably several prior papers show substantially more diverse skills. It seems that all the skills here are essentially pick and place skills, which is fine (these are interesting skills), but the title seems like a peculiar jab at prior work not being "diverse" enough, which is simply misleading.


FEW-SHOT LEARNING WITH GRAPH NEURAL NET- WORKS
We propose to study the problem of few-shot learning with the prism of inference on a partially observed graphical model, constructed from a collection of input images whose label can be either observed or not. By assimilating generic message-passing inference algorithms with their neural-network counterparts, we define a graph neural network architecture that generalizes several of the recently proposed few-shot learning models. Besides providing improved numerical performance, our framework is easily extended to variants of few-shot learning, such as semi-supervised or active learning, demonstrating the ability of graph-based models to operate well on 'relational' tasks.

This paper introduces a graph neural net approach to few-shot learning. Input examples form the nodes of the graph and edge weights are computed as a nonlinear function of the absolute difference between node features. In addition to standard supervised few-shot classification, both semi-supervised and active learning task variants are introduced. The proposed approach captures several popular few-shot learning approaches as special cases. Experiments are conducted on both Omniglot and miniImagenet datasets.
Strengths
- Use of graph neural nets for few-shot learning is novel.
- Introduces novel semi-supervised and active learning variants of few-shot classification.
Weaknesses
- Improvement in accuracy is small relative to previous work.
- Writing seems to be rushed.
The originality of applying graph neural networks to the problem of few-shot learning and proposing semi-supervised and active learning variants of the task are the primary strengths of this paper. Graph neural nets seem to be a more natural way of representing sets of items, as opposed to previous approaches that rely on a random ordering of the labeled set, such as the FCE variant of Matching Networks or TCML. Others will likely leverage graph neural net ideas to further tackle few-shot learning problems in the future, and this paper represents a first step in that direction.
Regarding the graph, I am wondering if the authors can comment on what scenarios is the graph structure expected to help? In the case of 1-shot, the graph can only propagate information about other classes, which seems to not be very useful.
Though novel, the motivation behind the semi-supervised and active learning setup could use some elaboration. By including unlabeled examples in an episode, it is already known that they belong to one of the K classes. How realistic is this set-up and in what application is it expected that this will show up?
For active learning, the proposed method seems to be specific to the case of obtaining a single label. How can the proposed method be scaled to handle multiple requested labels?
Overall the paper is well-structured and related work covers the relevant papers, but the details of the paper seem hastily written.
In the problem set-up section, it is not immediately clear what the distinction between s, r, and t is. Stating more explicitly that s is for the labeled data, etc. would make this section easier to follow. In addition, I would suggest stating the reason why t=1 is a necessary assumption for the proposed model in the few-shot and semi-supervised cases.
Regarding the Omniglot dataset, Vinyals et al. (2016) augmented the classes so that 4,800 classes were used for training and 1,692 for test. Was the same procedure done for the experiments in the paper? If yes, please update 6.1.1 to make this distinction more clear. If not, please update the experiments to be consistent with the baselines.
In the experiments, does the \varphi MLP explicitly enforce symmetry and identity or is it learned?
Regarding the Omniglot baselines, it appears that Koch et al. (2015), Edwards & Storkey (2016), and Finn et al. (2017) use non-standard class splits relative to the other methods. This should probably be noted.
The results for Prototypical Networks appear to be incorrect in the Omniglot and Mini-Imagenet tables. According to Snell et al. (2017) they should be 49.4% and 68.2% for miniImagenet. Moreover, Snell et al. (2017) only used 64 classes for training instead of 80 as utilized in the proposed approach. Given this, I am wondering if the authors can comment on the performance difference in the 5-shot case, even though Prototypical Networks is a special case of GNNs?
For semi-supervised and active-learning results, please include error bars for the miniImagenet results. Also, it would be interesting to see 20-way results for Omniglot as the gap between the proposed method and the baseline would potentially be wider.
Other Comments:
- In Section 4.2, Gc(.) is defined in Equation 2 but not mentioned in the text.
- In Section 4.3, adding an equation to clarify the relationship with Matching Networks would be helpful.
- I believe there is a typo in section 4.3 in that softmax(\varphi) should be softmax(-\varphi), so that more similar pairs will be more heavily weighted.
- The equation in 5.1 appears to be missing a minus sign.
Overall, the paper is novel and interesting, though the clarity and experimental results could be better explained.
EDIT: I have read the author's response. The writing is improved and my concerns have largely been addressed. I am therefore revising my rating of the paper to a 7.


Despite advances in deep learning, artificial neural networks do not learn the same way as humans do. Today, neural networks can learn multiple tasks when trained on them jointly, but cannot maintain performance on learnt tasks when tasks are presented one at a time -this phenomenon called catastrophic forgetting is a fundamental challenge to overcome before neural networks can learn continually from incoming data. In this work, we derive inspiration from human memory to develop an architecture capable of learning continuously from sequentially incoming tasks, while averting catastrophic forgetting. Specifically, our model consists of a dual memory architecture to emulate the complementary learning systems (hippocampus and the neocortex) in the human brain and maintains a consolidated long-term memory via generative replay of past experiences. We (i) substantiate our claim that replay should be generative, (ii) show the benefits of generative replay and dual memory via experiments, and (iii) demonstrate improved performance retention even for small models with low capacity. Our architecture displays many important characteristics of the human memory and provides insights on the connection between sleep and learning in humans.

This paper introduces a neural network architecture for continual learning. The model is inspired by current knowledge about long term memory consolidation mechanisms in humans. As a consequence, it uses:
-	One temporary memory storage (inspired by hippocampus) and a long term memory
-	A notion of memory replay, implemented by generative models (VAE), in order to simultaneously train the network on different tasks and avoid catastrophic forgetting of previously learnt tasks.
Overall, although the result are not very surprising, the approach is well justified and extensively tested. It provides some insights on the challenges and benefits of replay based memory consolidation.
Comments:

1-	The results are somewhat unsurprising: as we are able to learn generative models of each tasks, we can use them to train on all tasks at the same time, a beat algorithms that do not use this replay approach.
2-	It is unclear whether the approach provides a benefit for a particular application: as the task information has to be available, training separate task-specific architectures or using classical multitask learning approaches would not suffer from catastrophic forgetting and perform better (I assume).
3-	So the main benefit of the approach seems to point towards the direction of what possibly happens in real brains. It is interesting to see how authors address practical issues of training based on replay and it show two differences with real brains: 1/ what we know about episodic memory consolidation (the system modeled in this paper) is closer to unsupervised learning, as a consequence information such as task ID and dictionary for balancing samples would not be available, 2/ the cortex (long term memory) already learns during wakefulness, while in the proposed algorithm this procedure is restricted to replay-based learning during sleep.
4-	Due to these differences, I my view, this work avoids addressing directly the most critical and difficult issues of catastrophic forgetting, which relates more to finding optimal plasticity rules for the network in an unsupervised setting
5-	The writing could have been more concise and the authors could make an effort to stay closer to the recommended number of pages.


Published as a conference paper at ICLR 2018 MIXED PRECISION TRAINING
Increasing the size of a neural network typically improves accuracy but also increases the memory and compute requirements for training the model. We introduce methodology for training deep neural networks using half-precision floating point numbers, without losing model accuracy or having to modify hyperparameters. This nearly halves memory requirements and, on recent GPUs, speeds up arithmetic. Weights, activations, and gradients are stored in IEEE halfprecision format. Since this format has a narrower range than single-precision we propose three techniques for preventing the loss of critical information. Firstly, we recommend maintaining a single-precision copy of weights that accumulates the gradients after each optimizer step (this copy is rounded to half-precision for the forward-and back-propagation). Secondly, we propose loss-scaling to preserve gradient values with small magnitudes. Thirdly, we use half-precision arithmetic that accumulates into single-precision outputs, which are converted to halfprecision before storing to memory. We demonstrate that the proposed methodology works across a wide variety of tasks and modern large scale (exceeding 100 million parameters) model architectures, trained on large datasets.

The paper considers the problem of training neural networks in mixed precision (MP), using both 16-bit floating point (FP16) and 32-bit floating point (FP32). The paper proposes three techniques for training networks in mixed precision: first, keep a master copy of network parameters in FP32; second, use loss scaling to ensure that gradients are representable using the limited range of FP16; third, compute dot products and reductions with FP32 accumulation.
Using these techniques allows the authors to match the results of traditional FP32 training on a wide variety of tasks without modifying any training hyperparameters. The authors show results on ImageNet classification (with AlexNet, VGG, GoogLeNet, Inception-v1, Inception-v3, and ResNet-50), VOC object detection (with Faster R-CNN and Multibox SSD), speech recognition in English and Mandarin (with CNN+GRU), English to French machine translation (with multilayer LSTMs), language modeling on the 1 Billion Words dataset (with a bigLSTM), and generative adversarial networks on CelebFaces (with DCGAN).
Pros:
- Three simple techniques to use for mixed-precision training
- Matches performance of traditional FP32 training without modifying any hyperparameters
- Very extensive experiments on a wide variety of tasks
Cons:
- Experiments do not validate the necessity of FP32 accumulation
- No comparison of training time speedup from mixed precision
With new hardware (such as NVIDIA’s Volta architecture) providing large computational speedups for MP computation, I expect that MP training will become standard practice in deep learning in the near future. Naively porting FP32 training recipes can fail due to the reduced numeric range of FP16 arithmetic; however by adopting the techniques of this paper, practitioners will be able to migrate their existing FP32 training pipelines to MP without modifying any hyperparameters. I expect these techniques to be hugely impactful as more people begin migrating to new MP hardware.
The experiments in this paper are very exhaustive, covering nearly every major application of deep learning. Matching state-of-the-art results on so many tasks increases my confidence that I will be able to apply these techniques to my own tasks and architectures to achieve stable MP training.
My first concern with the paper is that there are no experiments to demonstrate the necessity of FP32 accumulation. With an FP32 master copy of the weights and loss scaling, can all arithmetic be performed solely in FP16, or are there some tasks where training will still diverge?
My second concern is that there is no comparison of training-time speedup using MP. The main reason that MP is interesting is because new hardware promises to accelerate it. If people are willing to endure the extra engineering overhead of implementing the techniques from this paper, what kind of practical speedups can they expect to see from their workloads? NVIDIA’s marketing material claims that the Tensor Cores in the V100 offer an 8x speedup over its general-purpose CUDA cores (https://www.nvidia.com/en-us/data-center/tesla-v100/). Since in this paper some operations are performed in FP32 (weight updates, batch normalization) and other operations are bound by memory and not compute bandwidth, what kinds of speedups do you see in practice when moving from FP32 to MP on V100?
My other concerns are minor. Mandarin speech recognition results are reported on “our internal test set”. Is there any previously published work on this dataset, or any publicly available test set for this task?
The notation around the Inception architectures should be clarified. According to [3] and [4], “Inception-v1” and “GoogLeNet” both refer to the architecture used in [1]. The architecture used in [2] is referred to as “BN-Inception” by [3] and “Inception-v2” by [4]. “Inception-v3” is the architecture from [3], which is not currently cited. To improve clarity in Table 1, I suggest renaming “GoogLeNet” to “Inception-v1”, changing “Inception-v1” to “Inception-v2”, and adding explicit citations to all rows of the table.
In Section 4.3 the authors note that “half-precision storage format may act as a regularizer during training”. Though the effect is most obvious from the speech recognition experiments in Section 4.3, MP also achieves slightly higher performance than baseline for all ImageNet models but Inception-v1 and for both object detection models; these results add support to the idea of FP16 as a regularizer.
Minor typos:
Section 3.3, Paragraph 3: “either FP16 or FP16 math” -> “either FP16 or FP32 math”
Section 4.1, Paragraph 4: “ pre-ativation” -> “pre-activation”
Overall this is a strong paper, and I believe that it will be impactful as MP hardware becomes more widely used.
References
[1] Szegedy et al, “Going Deeper with Convolutions”, CVPR 2015
[2] Ioffe and Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, ICML 2015
[3] Szegedy et al, “Rethinking the Inception Architecture for Computer Vision”, CVPR 2016
[4] Szegedy et al, “Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning”, ICLR 2016 Workshop


We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation. As a result, the gradient vanishing/explosion problem is avoided. Furthermore, we use the same analysis to discuss the tradeoff between depths and widths of a residual network and demonstrate that shallower yet wider resnets have stronger learning performance that deeper yet thinner resnets.

This paper attempts to analyze the gradient flow through a batchNorm-ReLU ResNet and make suggestions for reducing gradient explosion.
Firstly, the paper has a fatal mathematical flaw. Consider equation (10). There, you show the variance of y_{L,i} taken over BOTH random weights AND the batch. Now consider equation (32). In that equation, Var(y_{L,i}) appears in the denominator but this variance is taken over ONLY the batch and NOT the random weights. This Var(y_{L,i}) came from batch normalization, which divides its incoming activation values by their standard deviation. However, batch normalization only sees the variation in the activations given to it by a SPECIFIC set of weights. It does not know about the random variation of the weights because that randomness is in a sense a superstructure imposed on the network that the network operations themselves cannot see. Therefore, your substitution and therefore equation (13) is incorrect. If you replace the variance in equation (32) by the correct value, you will get a very different result from which very different (and very interesting!) conclusions can be drawn.
Secondly, in section 4, your analysis depends on the specific type of ResNet you chose. Specifically, when transitioning from one "scale" to the next, you chose to insert not just a convolutional layer, but also a batch normalization and ReLU layer on the residual path. To achieve scale transitions, in general, people use a single convolutional layer with 1*1 receptive field on the residual path. It is not a problem in itself to use a nonstandard architecture, but you do not discuss how your results would generalize to other ResNet architectures. Therefore your results have very limited relevance. (Note that again, of course, your results are corrupted by the variance problem I described earlier.)
Finally, with regards to section 5, let me be honest. (I hope that my area chair agrees with me that honesty is the best and kindest policy.) This section makes no sense. You do not understand the work by Veit et al. You do not know how to interpret gradient variances. While I won't be able to dicuss "gradient variance" as a concept in full detail in this review, here's a quick summary. (A) Veit et al. argued that a deep ResNet behaves as an ensemble of shallower networks as long as the gradient flowing through the residual paths is not larger than the gradient flowing through the skip paths. (B) The exploding gradient problem refers to the size of the gradient growing exponentially. The vanishing gradient problem refers to the size of the gradient shrinking exponentially. This can make it difficult to train the network. See "DEEP INFORMATION PROPAGATION" by Schoenholz et al. from ICLR 2017 to learn more about how gradient explosion can arise. (C) For a neural network to model a ground truth function exactly, the gradients of the network with respect to the input data have to match the gradients of the ground truth function with respect to the input. From observations (A) through (C), we can derive three guidelines for gradient conditioning: (A) have the gradient flowing through residual paths be not too small relative to the gradient flowing through skip paths, (B) have the gradient not grow or shrink exponentially with too large a rate, (C) have the data gradient match that of the ground truth function. However, you seem to be arguing that it is a problem if the gradient scale does increases too little from one residual block to the next. I am not aware of an established argument that this is indeed a problem. To be fair, one might make an argument as follows: "the point of deep nets is to be expressive, expressiveness of a layer relates to the spetrum of the layer-Jacobian, a small increase in gradient scale implies the layer-Jacobian has many similar singular values, therefore a small increase in gradient scale implies low expressiveness of the layer, therefore the layer is pathological". However, much more analysis, detail and care would be required to make this argument successfully. In any case, I also don't think that was the argument you were trying to make. Note that after I skimmed through the submissions to this conference, there seem to be interesting papers on the topic of gradients. Those papers plus the references provided in those papers should provide a good introduction to the topic of gradients in neural networks.
Other comments:
 - your notation is quite sloppy and may have lead to errors. Example: in the beginning of section 4, you say that from one "scale" to the next, the filter number increases k times. But in appendix C you say "Since the receptive field for the last scale is k times smaller". So is k the change in the receptive field size or the filter number of both? I would strongly recommend using dedicated variables to denote the width of the receptive field in each convolutional layer, the height of the receptive field in each convolutional layer as well as the filter number and then express all assumptions made in equation form.
- Equation (20) deals with the change of gradient variance within a scale. Where is the equation that shows the change of gradient variance between scales?
- I would suggest making all derivations in appendices A through D much more detailed.


A recent line of work has studied the statistical properties of neural networks to great success from a mean field theory perspective, making and verifying very precise predictions of neural network behavior and test time performance. In this paper, we build upon these works to explore two methods for taming the behaviors of random residual networks (with only fully connected layers and no batchnorm). The first method is width variation (WV), i.e. varying the widths of layers as a function of depth. We show that width decay reduces gradient explosion without affecting the mean forward dynamics of the random network. The second method is variance variation (VV), i.e. changing the initialization variances of weights and biases over depth. We show VV, used appropriately, can reduce gradient explosion of tanh and ReLU resnets from exp(Θ( √ L)) and exp(Θ(L)) respectively to constant Θ(1). A complete phase-diagram is derived for how variance decay affects different dynamics, such as those of gradient and activation norms. In particular, we show the existence of many phase transitions where these dynamics switch between exponential, polynomial, logarithmic, and even constant behaviors. Using the obtained mean field theory, we are able to track surprisingly well how VV at initialization time affects training and test time performance on MNIST after a set number of epochs: the level sets of test/train set accuracies coincide with the level sets of the expectations of certain gradient norms or of metric expressivity (as defined in Yang and Schoenholz (2017)), a measure of expansion in a random neural network. Based on insights from past works in deep mean field theory and information geometry, we also provide a new perspective on the gradient explosion/vanishing problems: they lead to ill-conditioning of the Fisher information matrix, causing optimization troubles.

Mean field theory is an approach to analysing complex systems where correlations between highly dependent random variables are ignored, thus making the problem analytically tractable. It is hoped that analytical insights gained in this idealised setting might translate back to the original (and far messier) problem. The authors use a mean field theory approach to study how varying certain network hyperparameters with depth can effect gradient and activation statistics. A correlation between the behaviour of these statistics and training performance on MNIST is noted.
As someone asked to conduct an 'emergency' review of this paper, I would have greatly appreciated the authors making more of an effort to present their results clearly. Some general comments in this regard:
Clarity issues:
- the authors appear to have ignored the ICLR style guidelines
- the references are all written in green, making them difficult to read
- figures are either missing color maps or make poor choice of colors
- the figure captions are difficult to understand in isolation from the main text
- the authors themselves appear to muddle their 'zigs' and 'zags' (first line of discussion)
Now to get to the actual content of the paper. The authors do not properly place their work in context. Mean field theory has been studied in the context of neural networks at least since the 80's. Entire books have been written on the statistical mechanics of neural networks. It seems wrong that the authors only cite papers on this matter going back to 2016.
With that said, the main thrust of the paper is very interesting. The authors derive recurrence relations for mean activations and gradients. They show how scaling layer width and initialisation variance with depth can better control the propagation of these means. The results of their calculations appear to match their random network simulations, and this part of the work seems strong.
What is not clear is what effect we should expect these quantities to have on learning? The authors claim there is a tradeoff between expressivity and exploding gradients. This seems quite speculative since it is not clear to me what effect either of these things will have on training. For one, how expressive does a model need to be to correctly classify MNIST? And are exploding gradients necessarily a bad thing? Provided they do not reach infinity, can we not just choose a smaller learning rate?
I'm open to reevaluating the review if the issues of clarity and missing literature review are fixed.


Published as a conference paper at ICLR 2018 CONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION
Social dilemmas, where mutual cooperation can lead to high payoffs but participants face incentives to cheat, are ubiquitous in multi-agent interaction. We wish to construct agents that cooperate with pure cooperators, avoid exploitation by pure defectors, and incentivize cooperation from the rest. However, often the actions taken by a partner are (partially) unobserved or the consequences of individual actions are hard to predict. We show that in a large class of games good strategies can be constructed by conditioning one's behavior solely on outcomes (ie. one's past rewards). We call this consequentialist conditional cooperation. We show how to construct such strategies using deep reinforcement learning techniques and demonstrate, both analytically and experimentally, that they are effective in social dilemmas beyond simple matrix games. We also show the limitations of relying purely on consequences and discuss the need for understanding both the consequences of and the intentions behind an action.

This paper studies learning to play two-player general-sum games with state (Markov games) with imperfect information. The idea is to learn to cooperate (think prisoner's dilemma) but in more complex domains. Generally, in repeated prisoner's dilemma, one can punish one's opponent for noncooperation. In this paper, they design an apporach to learn to cooperate in a more complex game, like a hybrid pong meets prisoner's dilemma game. This is fun but I did not find it particularly surprising from a game-theoretic or from a deep learning point of view.
From a game-theoretic point of view, the paper begins with a game-theoretic analysis of a cooperative strategy for these markov games with imperfect information. It is basically a straightforward generalization of the idea of punishing, which is common in "folk theorems" from game theory, to give a particular equilibrium for cooperating in Markov games. Many Markov games do not have a cooperative equilibrium, so this paper restricts attention to those that do. Even in games where there is a cooperative solution that maximizes the total welfare, it is not clear why players would choose to do so. When the game is symmetric, this might be "the natural" solution but in general it is far from clear why all players would want to maximize the total payoff.
The paper follows with some fun experiments implementing these new game theory notions. Unfortunately, since the game theory was not particularly well-motivated, I did not find the overall story compelling. It is perhaps interesting that one can make deep learning learn to cooperate with imperfect information, but one could have illustrated the game theory equally well with other techniques.
In contrast, the paper "Coco-Q: Learning in Stochastic Games with Side Payments" by Sodomka et. al. is an example where they took a well-motivated game theoretic cooperative solution concept and explored how to implement that with reinforcement learning. I would think that generalizing such solution concepts to stochastic games and/or deep learning might be more interesting.
It should also be noted that I was asked to review another ICLR submission entitled "MAINTAINING COOPERATION IN COMPLEX SOCIAL DILEMMAS USING DEEP REINFORCEMENT LEARNING" which amazingly introduced the same "Pong Player’s Dilemma" game as in this paper.
Notice the following suspiciously similar paragraphs from the two papers:
From "MAINTAINING COOPERATION IN COMPLEX SOCIAL DILEMMAS USING DEEP REINFORCEMENT LEARNING":
We also look at an environment where strategies must be learned from raw pixels. We use the method
of Tampuu et al. (2017) to alter the reward structure of Atari Pong so that whenever an agent scores a
point they receive a reward of 1 and the other player receives −2. We refer to this game as the Pong
Player’s Dilemma (PPD). In the PPD the only (jointly) winning move is not to play. However, a fully
cooperative agent can be exploited by a defector.
From "CONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION":
To demonstrate this we follow the method of Tampuu et al. (2017) to construct a version of Atari Pong
which makes the game into a social dilemma. In what we call the Pong Player’s Dilemma (PPD) when an agent
scores they gain a reward of 1 but the partner receives a reward of −2. Thus, in the PPD the only (jointly) winning
move is not to play, but selfish agents are again tempted to defect and try to score points even though
this decreases total social reward. We see that CCC is a successful, robust, and simple strategy in this
game.