We consider the following central question in the field of Deep Reinforcement Learning (DRL): How can we use implicit human feedback to accelerate and optimize the training of a DRL algorithm? State-of-the-art methods rely on any human feedback to be provided explicitly, requiring the active participation of humans (e.g., expert labeling, demonstrations, etc.). In this work, we investigate an alternative paradigm, where non-expert humans are silently observing (and assessing) the agent interacting with the environment. The human's intrinsic reactions to the agent's behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials. The implicit feedback is then used to augment the agent's learning in the RL tasks. We develop a system to obtain and accurately decode the implicit human feedback (specifically error-related event potentials) for state-action pairs in an Atari-type environment. As a baseline contribution, we demonstrate the feasibility of capturing error-potentials of a human observer watching an agent learning to play several different Atari-games using an electroencephalogram (EEG) cap, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm with the intent of accelerating its learning of the game. Building atop the baseline, we then make the following novel contributions in our work: (i) We argue that the definition of error-potentials is generalizable across different environments; specifically we show that error-potentials of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the error-potentials. (ii) We propose two different frameworks to combine recent advances in DRL into the error-potential based feedback system in a sample-efficient manner, allowing humans to provide implicit feedback while training in the loop, or prior to the training of the RL agent. (iii) Finally, we scale the implicit human feedback (via ErrP) based RL to reasonably complex environments (games) and demonstrate the significance of our approach through synthetic and real user experiments.

This paper tackles the problem of obtaining feedback from humans for training RL agents, in a way that does not require too much time or mental burden. The proposed approach does this by using implicit human feedback, obtained by measuring EEG signals from the brains of human observers. When humans spot an error (e.g., in the behaviour of the RL agent), an error potential (ErrP) is generated, which differs from human to human. The proposed approach first uses supervised learning to learn what the ErrP looks like for a particular person, and then uses this to provide an additional reward signal to the agent -- the agent receives a negative reward for state-action pairs that result in an ErrP.
The goal of this paper is to show that using implicit human feedback via ErrPs is a feasible way to speed up the training of RL agents. Towards this, the paper makes two main contributions: (1) experimental evidence that the ErrP for a human can be learned for one game, and transferred zero-shot to other games, and (2) two approaches of collecting this implicit feedback from humans for a smaller number of trajectories, instead of collecting it for all trajectories encountered by the agent during learning. The experimental evaluations, on three simple Atari-like domains, show that agents learn more quickly with implicit feedback, even when gathered for a small number of trajectories, compared to not having any feedback.
This paper is well-motivated, and tackles an important problem. However, I have several concerns related to feasibility and reproducibility, as described below. The writing also needs quite a bit of editing; there are typos, grammatical errors, and incorrect / missing figure and section references.
Regarding feasibility, my first concern is that the domains considered here are relatively simple. Is there evidence that learned ErrPs would also generalize between tasks with more complex visuals? Also, in order to obtain these error signals, agent trajectories need to be slowed down significantly (from multiple actions a second to one action every 1.5 seconds). I imagine this would quickly become very tedious for people. This seems to be an inherent limitation of this approach, because relevant EEG signals happen up to 1+ seconds after an event occurs. I have doubts that this would generalize to more complex, long-horizon tasks, in which trajectories are at least hundreds of timesteps long. Finally, the study was conducted on a small number of participants (five), with a narrow range of ages and limited gender diversity. I sympathize with the difficulty in obtaining subjects for such studies, but the small sample size makes it questionable whether these results on transferability of learned ErrPs would apply to the wider population.
Reproducibility questions:
- How are the labels (of error vs. not error) obtained for learning per-person error potentials? For more complex domains, it seems that it would be harder to obtain these ground-truth labels, because if there are substantially different strategies for playing the game, humans could disagree on which state-action pairs they consider to be errors.
- In Section 4.2, how exactly does the RL agent use the saved state-action error labels (from queries to the human) to determine if the current state-action pair is an error? Is it using some form of nearest-neighbor? The notion of buckets isn't clearly described.
Additional questions and comments:
- Why does Wobble have the worst AUC for learning ErrPs (Figure 5a)? It seems to be the game that has the most obvious errors (i.e., agent moves away from the target), so I would expect it to have the best AUC.
- In Section 5.2.1 (Generalizability), what are hypotheses for *why* the ErrP trained on Wobble generalizes better to certain situations in Catch and worse for others? It would be useful to be able to characterise / predict how good generalization will be from one type of game / situation to another.
- I would like to see reports of wall-clock time for collection of implicit human feedback, for full access and the two other approaches. In other words, how long do humans have to spend connected to the electrodes?
- I'm curious how imperfect the demonstrations need to be, in order for the ErrPs for those demonstrations to provide a useful reward signal to the agent. In the paper, incorrect actions appear with a probability of 0.2, but there's no explanation for how this number was selected, and whether others were tested.
Minor comments:
- The diagram in Figure 2a is confusing because DQN is there twice —- explicitly as a component, and implicitly as part of the RL agent.
- The x-axis labels in Figure 5b don't agree with the subject labels in Figures 4b and 7b.
- In Figure 6, data for only two of the five participants is reported; it would be useful to include the results for the other participants in the Appendix.


Generative models, such as GANs, learn an explicit low-dimensional representation of a particular class of images, and so they may be used as natural image priors for solving inverse problems such as image restoration and compressive sensing. GAN priors have demonstrated impressive performance on these tasks, but they can exhibit substantial representation error for both in-distribution and out-of-distribution images, because of the mismatch between the learned, approximate image distribution and the data generating distribution. In this paper, we demonstrate a method for reducing the representation error of GAN priors by modeling images as the linear combination of a GAN prior with a Deep Decoder. The deep decoder is an underparameterized and most importantly unlearned natural signal model similar to the Deep Image Prior. No knowledge of the specific inverse problem is needed in the training of the GAN underlying our method. For compressive sensing and image superresolution, our hybrid model exhibits consistently higher PSNRs than both the GAN priors and Deep Decoder separately, both on in-distribution and out-of-distribution images. This model provides a method for extensibly and cheaply leveraging both the benefits of learned and unlearned image recovery priors in inverse problems.

Summary: This paper proposes to use a combination of a pretrained GAN and an untrained deep decoder as the image prior for image restoration problem. The combined model jointly infers the latent code for the trained GAN and the parameters in the untrained deep decoder. It also jointly infers the mixing coefficient alpha and beta during test time for each image, thus learning how much we should rely on GAN. The proposed hybrid model is helpful on compressed sensing experiments on the CelebA dataset; however, it is only marginally better than deep decoder on image super resolution and out-of-distribution compressed sensing.
Detailed comments:
-	The writing is clear and I was able to understand the model part of the paper. The algorithm box is helpful. However, I would still appreciate if the authors can provide an overall model figure in the model section to help understanding.
-	Jointly learning the mixing coefficient is an interesting part of the model.
-	The motivation in the abstract and intro could be strengthened. A smaller version of Figure 1 can be probably moved to the beginning of the paper to illustrate the problem of GAN. But even with the help of Figure 1, it is still unclear what is the fundamental problem for GAN. Simply combining a GAN with an untrained decoder model doesn’t help elucidate the source of the problem.
-	The proposed Hybrid model seems to help on compressed sensing experiments on CelebA. However, it doesn’t help much on out-of-distribution experiments. Moreover, in the super-resolution task, as shown in Figure 5, the improvement over deep decoder is also not significant.
-	The out-of-distribution experiments seems lack of thorough study. In particular, the paper only studies the transfer between CelebA -> Caltech-UCSD Bird dataset. It would be better if the paper can study a variety of other image datasets as well. Also some visualization on the Bird dataset should also be included.
-	Effect of n_pre needs to be further investigated. Why not directly train both models together? It would be good if the authors could comment on how sensitive the n_pre is and what is the intuition.
-	For figures, I would recommend rename “Hybrid” to “Hybrid (Ours)” to highlight the paper’s contribution, and use a brighter color.
-	Figure 5 should be renamed as a Table.
-	Hyperparameter details should be moved to the Experiment section.
Conclusion:
The paper proposes a simple combination of a trained GAN and an untrained decoder model for the task of image restoration. Although the method is clear and straightforward, in the experiments, the influence of the new model component seems marginal. Moreover, the motivation is not strong enough. Therefore, I recommend weak reject.


In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, much recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

This paper proposes a unifying framework, BRAC, which summarizes the idea and evaluates the effectiveness of recently proposed offline reinforcement learning algorithms, specifically BEAR, BCQ, and KL control. The authors generalize existing offline RL approaches to an actor-critic algorithm that regularizes the learned policy such that it can stay close to the behavior policy. Based on prior work, the authors state that there are two variants of regularizations to the behavior policy, value penalty (vp) and policy regularization (pr) and three choices of divergence functions along with their sample estimate that measures the distance between the learned policy and the behavior policy. The paper empirically investigates the effectiveness of each regularization scheme as well as each divergence function and conclude that vp is slightly more effective than pr while all divergence functions have similar performances.
Overall, this paper could be an interesting summary of prior works in offline RL and provide some empirical insights on the effectiveness of each building block in the previous approaches, though it neither offers theoretical explanations nor proposes a new offline RL algorithm that outperforms the existing methods under the BRAC framework. My score would be increased given some technical insights or some promising results in a relatively novel offline RL algorithm in the author’s response.
Specifically, the paper does a thorough ablation study on BEAR, BCQ, and KL-control within the BRAC framework. While the experimental results demonstrate some interesting phenomenons such as combining vp and the primal form of KL divergence achieving the best performance and taking minimum over Q functions outperforming using a mix of maximum and minimum over the Q functions, I believe the paper would be greatly improved if the authors can provide a new offline RL method based on the BRAC that can achieve better performance than current approaches and is less incremental than simply combining vp and KL divergence.
Though BRAC summarizes offline RL methods in a neat way, it would be more technically sound if a general theoretical analysis/insights of offline RL algorithms can be offered in the paper, e.g. showing the reason that vp is outperforming pr through convergence analysis in the tabular case.
Minor comment:
Error bars should be added to the all the bar plots.
**UPDATE**
After reading the rebuttal, I think my concerns are addressed and thus I updated my rating.


We release the largest public electrocardiogram (ECG) dataset of continuous raw signals for representation learning containing 11 thousand patients and 2 billion labelled beats. Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events. To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion. We provide a set of baselines for different feature extractors that can be built upon. Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

Overall the paper is well written and organized and is an application of various paradigms to extract most relevant features for Arrhythmia Subtype classification.
Plan to release the largest (claimed) public ECG data set of continuous raw signals for representation learning containing 11 thousand patients and 2 billion labelled beats.
Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown sub types of arrhythmia and anomalous ECG signal events.
The stated intended goal though is the discovery of new  Arrhythmia Sub types (title), “automated arrhythmia detection” (1.1 Objective).
A definition/example of a regular ECG signal, one presenting arrhythmia, various types of arrhythmia signal forms, would be welcome for the paper to fit this audience and be self-contained.
This work suffers from many drawbacks:
-	A definition/example of a regular ECG signal, one presenting arrhythmia, various types of arrhythmia signal forms, would be welcome for the paper to fit this audience and be self-contained.
-	Given the intended audience in ICLR, the application domains specificities are not well explained. Many concepts are not introduced clearly:
      o	ECG models, sub types of arrhythmia, anomalous ECG signal events.
      o	In related work section:
            	Paragraph 2: more “leads”, RR interval, PR interval, QRS duration, induced myocardial ischemia.
            	Paragraph 3: single-lead wearable devices.
      o	In 3 privacy concerns: heartbeats as biometrics section/ Paragraph 2:
            	alternative ways to sense “cardiac motion”.
      o	In 4 {companyname} 11k data set section:
            	Paragraph 2: “third line exam”.
            	Paragraph 3: “beats and rhythms”, these are fundamental to understanding graphical results provided.
      o	In 5.1 quantitative evaluation section:
            	Paragraph 3: “ .. irregular RR intervals, no distinct P waves and usually variable intervals between two atrial activations.
            	Page 6:  “ .. high-level abnormality labels”
      o	5.2 qualitative evaluation section
            	Paragraph 1:  “ .. has a dominant S wave in V1 lead.”
      o	In 6 conclusion section:
            	“ Single-lead heart monitors” .
-	Figures are barely decipherable
      o	In Figure 6:
            	What are ESSV and ESV, btype and rtype. Could you give a list of known subtypes?
            	Is this for PVC data only?
-	Many paragraphs are not clear to us:
      o	Introduction/Paragraph 2:
            	“While cardiologists are able to see these differences, it is hard to conclude that they are ‘real’ by finding the same anomalous signal across multiple time points and patients without a data driven approach.”
      o	In 3 privacy concerns: heartbeats as biometrics section/ Paragraph 3:
            	Contrary to 1), 3) & 4), is (2) “.. the expression of environmental variables on the heartbeat data is unique to the individual”, a limitation to ECG to being considered a biometric measure?
      o	In 4 {companyname} 11k dataset section:
            	Paragraph 4:  “we segment each patient record into segments of 220 +1 signal samples ( ~70 minutes). Care to explain the rationale?
            	Last paragraph: “ .. we believe that processing the data with these levels of hierarchy results in some grouping information that could be leveraged to attain better results.”. This is a multi-scale approach. But do you have any medical (application domain) knowledge that would justify/hint to using such approach?
      o	Figure 5:
            	Only Beats are labelled. No Rhythm labelling?
      o	Table 2:
            	“Only 2 types of labels are provided”. Are these beats and Rhythms? What are anomalies considered in the study then?
      o	In 5.1 quantitative evaluation section:
            	Paragraph 2: “..PAC is an abnormal beat only because it appears too soon and disrupts the rhythm (frequency). Furthermore, a PAC beat has the same shape as a normal beat, so taken alone, you can nearly not make the difference with a normal beat.”. A graphical representation would be welcome.
            	Paragraph 3: “..Both require a representation that will compose a representation showing the difference between beats over time.”.
      o	qualitative evaluation section
            	Paragraph 2: “We note that these are easy to see because of the different colors we use to highlight the points, but there seems to be remaining clusters that have not been analysed.”. The clusters are not evident to us even after purposefully colored.


While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Adam have been observed to outperform SGD across important tasks, such as attention models. The settings under which SGD performs poorly in comparison to Adam are not well understood yet. In this paper, we provide empirical and theoretical evidence that a heavy-tailed distribution of the noise in stochastic gradients is a root cause of SGD's poor performance. Based on this observation, we study clipped variants of SGD that circumvent this issue; we then analyze their convergence under heavy-tailed noise. Furthermore, we develop a new adaptive coordinate-wise clipping algorithm (ACClip) tailored to such settings. Subsequently, we show how adaptive methods like Adam can be viewed through the lens of clipping, which helps us explain Adam's strong performance under heavy-tail noise settings. Finally, we show that the proposed ACClip outperforms Adam for both BERT pretraining and finetuning tasks.

The paper proposed a very interesting claim: When training a neural network, if the (stochastic) gradient noise is Gaussian-like, then SGD performs better than Adam; On the other hand if the gradient noise is Heavy tailed, then Adam perform better than SGD.
The paper supported this argument with experiments showing that ResNet50 on ImageNet, the noise is more like Gaussian while BERT on language learning tasks the noise is more heavy-tailed. The paper also gave a theoretical result showing that Adam converges in the regime of heavy-tailed noise.
The experiment finding is quite surprising to me, since many papers (see e.g.
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
)  claim that the SGD noise is heavy-tailed for image recognization tasks such as CIFAR-10, CIFAR-100. The referred paper used rigorous statistical testing for the tail-index of the SGD noise, while this paper simply drew some image.
Moreover, the theoretical result in this paper also worries me quite a bit, since from the bounds it seems that Adam is the dominating algorithm (both in the heavy-tail case and in Gaussian tail case). Moreover, SGD also converges in the heavy-tail noise case (by showing that the norm of x_t is not too large during the training process using martingale-based argument). Hence, the upper bound of the theoretical result is convincing enough to claim that Adam is better than SGD in certain regime.
After Rebuttal: I have read the authors' responses and acknowledge the sensibility of the statement. I have higher my score: In particular, if the noise is indeed Gaussian as opposite to the "known results", this paper should be accepted.


This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.

The paper addresses the problem of exploiting human demonstrations in hard exploration (RL) problems. A new set of challenge tasks is introduced that destroys the performance of very strong baseline systems while highlighting the strength of the new system.
The approach (rarely but consistently training on separately prioritized human experience replays) is well motivated by the shortcomings of past agents (either in overfitting the demonstrated solution or only working in environments with not-too-hard exploration challenges). Where work by others have overspecialized on specific challenge environments (e.g. Montezuma's Revenge with weak stochasticity and observability challenges), this work intentionally dives into difficult territory.
This reviewer moves to accept this top-quality RL paper. The new agent, R2D3, is the primary contribution in combining and outperforming previous SOTA agents. These eight new environments are minor contributions with limited potential for impact on the field, but still make an independently positive contribution.
Questions for authors:
- What would be the present-day approximate retail cost for reproducing the experiments in this paper?
- At the action-rate experienced by the human demonstrators (30fps?), how much wall-clock time represented by 40B actor steps? (40 years?) Is this "making efficient use"?
- Does having highly variable initial conditions really force generalization over environmental configurations or is this wishful thinking / mysticism? To make a direct claim about this, the authors should consider an experimental design where certain classes of initial conditions (e.g. starting on the left side of the map) are withheld during training and evaluated only during testing.
- The finding of small demo ratios as being stronger is exciting, but this result seems to be tied to the specific quantity and quality of demonstrations gathered. Could a more general picture of the role of demonstrations be had by ablating the diversity of representations? The 100 demos in the full case might be degraded to 50, 25, 10, etc while holding the demo ratio fixed. This might effectively vary the weight that demonstrations take in the optimization independently of how often distinct demos are actually seen.
- Can these hard-eight scenarios be parametrically scaled up and down in terms of their exploration effort (possibly by just changing the action granularity / movement speed)? With performance on the new benchmark almost saturated in the first paper based on it, there isn't much room to grow here. In the same way that Montezuma's Revenge was found by scanning the culturally-impactful library of Atari games, perhaps more appropriate and lasting challenges can be found by looking one or more generations forward in the history of commercial console games. Can we play Star Fox? What about SimCity?


Neural Networks (NNs) have been extensively used for a wide spectrum of realworld regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that prediction must also be estimated. Standard NNs, which are most often used in such tasks, do not provide uncertainty information. Existing approaches address this issue by combining Bayesian models with NNs, but these models are hard to implement, more expensive to train, and usually do not predict as accurately as standard NNs. In this paper, a new framework (RIO) is developed that makes it possible to estimate uncertainty in any pretrained standard NN. The behavior of the NN is captured by modeling its prediction residuals with a Gaussian Process, whose kernel includes both the NN's input and its output. The framework is justified theoretically and evaluated in twelve real-world datasets, where it is found to (1) provide reliable estimates of uncertainty, (2) reduce the error of the point predictions, and (3) scale well to large datasets. Given that RIO can be applied to any standard NN without modifications to model architecture or training pipeline, it provides an important ingredient for building real-world NN applications.

# Summary
The authors propose a method for post-hoc correction and predictive variance estimation for neural network models. The method fits a GP to the model residuals, and learns a composite kernel that combines two kernels defined on the input space and the model’s output space (called RIO, R for residual, and IO for the input-output kernel). The authors suggest that residual means and variances at test points can then be calculated explicitly using predictive distributions from the GP. The authors run a large panel of experiments across a number of datasets, and compare to a number of methods that draw connections between neural networks and GP’s. In addition, the full method is compared to a number of methods that utilize only some components of the full RIO method. In these experiments, the RIO method generally shows strong performance in both RMSE and NLPD compared to these baselines.
# Feedback
Overall, this is a neat method. It has the flavor of a number of other composite ML methods that have worked well in the past---e.g., boosting and platt scaling---but is different enough to stand on its own. The experimental results are quite promising.
However, I am torn about the paper, because the theoretical discussion of the method is quite convoluted and seems either irrelevant or incorrect. I wish that the authors had spent more time with small demonstrations of what the procedure does in some simple settings. This would give practitioners considering the method far more intuition about when they would expect it to work and fail than the current theoretical discussion.
## Uncertainty Discussion is Lacking
The motivation and discussion sell this method as an uncertainty quantification method, but almost all of the theoretical development revolves around prediction correction. The methods properties as an uncertainty quantification tool are underdeveloped.
The only theoretical point made about uncertainty estimation is Theorem 2.7, which states that the scalar variance of the GP “nugget” is positively correlated with the variance of the NN’s residuals. Providing a scalar summary of noise is not particularly compelling for a method advertised as a point-prediction uncertainty quantification method. In addition, it is not clear what probability distribution the “correlation” is defined over. The argument made in the proof seems quite obvious: if a GP is used to model a noisier process (i.e., residuals with a larger variance), it will in some cases classify that variability as independent noise.
If the authors wanted to focus on the properties of their method as an uncertainty quantification tool, they could discuss the assumptions underlying the GP error estimates, and when they would be likely to diverge from practical properties like predictive interval coverage. For example, because the base NN predictor is treated as fixed, it seems that this method ignore uncertainty that stems from the NN fit due to random initialization. Likewise, it seems that this method would not quantify uncertainty from resampling the data and obtaining a new NN predictor. )The coverage experiments in the appendix seem to confirm this -- generally, the predictive intervals generally under-cover the predicted values.) It’s fine if the method doesn’t quantify these types of uncertainty, but discussion of these types of issues would be far more welcome than the current convoluted theory in Section 2. This discussion might not yield theorems, but it would give practitioners useful guidelines for deciding whether the particular scheme would likely work for this application.
## Problems with the Error Correction Argument
The theory section, especially 2.2, was very difficult to parse. First, as a matter of style, a sequence of Lemma and Theorem statements are given without defining most of the notation used therein, and with almost no prose providing context or intuition. In the buildup to the theorems, it is also unclear which assertions about the decompositions of y_i are assumptions about the true data generating process, and which assertions are specifications of a particular GP model.
The substance also has some issues. I think the intention in this section is to get to a rather simple variance decomposition of the labels y. The question is how much variation in y or the residual is represented in the posterior predictive mean of a particular GP. It seems reasonable that in some cases, the structure in the residual may be more amenable to modeling with a stationary GP than the structure in the raw labels y. It is not clear that all of the theoretical complexity here is necessary to make this point.
Instead, the authors make a convoluted argument that attempts to establish that the errors from the NN + GP approach will be smaller under very general circumstances. The argument is phrased somewhat ambiguously (it is not clear exactly what is being assumed, and what is corresponds to the specification of a working model), but depending on how one reads this section, the argument makes statements that are either too broad to be correct, or too narrow to be relevant.
The argument decomposes for the raw labels and the residuals into pieces that a GP can “capture” or “represent”, and parts that it cannot. The two equations are:
y_i = f(x_i) + g(x_i) + \xi_i
R_i = (y_i - h_NN(x_i)) = r_f(x_i) + r_g(x_i) + \xi_i
f(.) and r_f(.) represent the portions of the label and residual processes, respectively, that the GP "captures". It is assumed that the GP will model this portion correctly, and leave the “epsilon-indistinguishable” portion g(.) or r_g(.) untouched. The argument then assumes that f(.) and r_f(.) will have proportional kernels, and so it is possible to show that the predictions of residuals based on r_f(.) will have smaller predictive variance than predictions based on f(.) as long as the variation represented by r_g(.) is smaller than the variation represented by g(.).
On its face, this argument raises some red flags. Because h_NN(.) is allowed to be an arbitrary function, the argument here should be symmetric. Why can’t we also get a guaranteed variance reduction by adding h_NN(.) to y rather than subtracting it? Perhaps some of this is captured in the parameter \delta, which quantifies the reduction in variation represented in r_g(.) vs g(.), but the argument that the kernel of r_f(.) can be no larger than the kernel f(.) in terms of trace (that is, the proportionality constant \alpha is not greater than 1) does not make sense. If h_NN(x_i) is simply -f(x_i), then these arguments would not go through. At the very least, conditions need to be articulated about the properties of h_NN(.).
Some of the strangeness comes from the fact that this is a poor model of most prediction problems, where the main issue with fitting a GP is not “indistinguishability”, but misspecification. Consider a process y_i that is non-stationary; say g(.) has a linear trend in some component of x. A GP with a stationary covariance kernel fit to this process (such as RBF) will attempt to explain the variation due to the linear trend with a variance kernel that encodes long-range dependence. On the other hand, if this trend were removed by a base model like an NN, the residuals would have a very different structure (perhaps they would be stationary), and in this case, the GP would fit the data with a very different covariance kernel.
Unfortunately, it does not seem like the formalism here can express a notion of misspecification at all. In the theory, it is assumed that the GP will only model the portion of the labels y_i for which it is property specified (in this case, f(.)). This generally does not occur in practice, as in the example above. It might be possible for this to apply in some circumstances, but the authors give no conditions (e.g., that the process y_i be stationary). Based on this assumption, the authors assert that the fitted GP to f(.) and r_f(.) will have the same covariance kernel parameters up to some proportionality constant \alpha. Much of their theoretical argument depends on this proportionality. But this proportionality cannot apply in general, and again, no conditions are given for when we might expect this to hold.
It would be far more compelling if the authors proposed the very standard approach to modeling data via covariance kernels, where one first models non-stationary portions of the data with a base model, then models the correlation in the residuals with something like a GP. This is the bread-and-butter approach in, say, timeseries analysis (see, e.g., the Shumway and Stoffer textbook https://www.stat.pitt.edu/stoffer/tsa4/tsa4.htm), and the approach in this paper could be framed similarly.
## Demos I Wish I Had Seen
I wish the authors had presented some demonstrations of what the GP does to the fitted values of an NN. Giving a demonstration of how the output kernel modifies predicted values, for example, would give some nice intuition the value added by this portion. I suspect that this step essentially performs something like Platt scaling, but for continuous outcomes, by shrinking predictions together so that they better match the overall distribution of observed labels. Perhaps the mechanism is different. At any rate, it would be useful to understand where the information gain is coming from, and this would be far better expressed concretely in terms of a toy data example than the theoretical arguments that are given.
## Coverage Experiments
I wish the coverage experiments evaluating predictive intervals were included in the main text. As far as uncertainty quantification evaluations go, coverage is one of the few assessments that does not rely on the model itself (unlike NPLD, which uses the model’s own log-likelihood), and can be phrased as a concrete performance guarantee.
Here, the goal for predictive intervals is to cover the true prediction value _at least as often_ as the nominal rate (95% intervals should cover the truth _at least_ 95% of the time), not merely that coverage be “close” to the nominal rate. This asymmetric evaluation gives you a concrete guarantee that the uncertainty estimate is conservative. The coverage experiments show that this method quite systematically under-covers compared to the end-to-end SVGP method, which generally satisfies this coverage property. I think this is important information to include about the model, and generally I think this behavior results from the fact that uncertainty is not propagated from the NN fit. This should be presented clearly in the main text.


This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources. We design a Dynamic Point-cloud Convolution (DConv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input. This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -an important aspect in spatiotemporal predictive learning. The DConv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms. We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e., mobile service traffic forecasting and air quality indicator forecasting. Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of neural network models.

Review Summary
--------------
Overall this is almost above the bar for me to accept, but I think there's enough concerns about the method and experiments that I'm hesitant. Strengths include the invariance to point cloud order and the relative simplicity of the architecture (compared to PointCNN). Weaknesses include a vulnerability to outliers, experiments that don't seem to think about practical effects like day-of-week in forecasting, and experiments that leave out baselines to help directly assess the impact of neighbors.
Paper Summary
-------------
The paper develops a new neural net architecture for processing data structured as spatial point clouds that vary over time (e.g. hourly traffic at several antennas spread throughout a city).
The core of the approach is a new neural net unit: the "D-Conv" operator (See Eq. 2). The output value at each point is obtained via a weighted combination of nearby coordinates and features, using only the K-nearest neighbors (stored in ranked order) to maintain invariance to the original order of points. This layer can be included in modern convolutional (CloudCNN) or recurrent (CloudLSTM) or attention-based architectures in a straightforward way.
Unlike many previous methods that require converting point clouds to quantized regular grids, the present approach directly consumes point cloud data. Unlike some existing methods like PointCNN, it avoids information loss (does not reduce dimension from input to output layer).
Two experimental evaluations are conducted: forecasting mobile app traffic across 2 European cities (given past 30 min, predict next 30 min), and air quality across several regions in China (given last 12 hrs, predict next 12 hrs). In both experiments, the locations of the sensors are fixed across time. Fig 4 further looks at traffic forecasting as a function of the lookahead time, from 0-3 hours ahead.
Novelty & Significance
-----------------------
The paper definitely tackles an important problem (point cloud forecasting).
The present paper's new "D-Conv" operator appears new, though it looks really like a simplification of the PointCNN's "X-Conv" operator rather than a brand new operator.
The most similar work seems to be the PointCNN (Li et al NeurIPS 2018). This work's contribution was a new "X-Conv" operator, which also consumes point clouds and produces learned representations. X-Conv, like the present paper's D-Conv, computes K-nearest neighbors of each point p, but performs first an embedding of each neighbor to a learned "local" feature space and then performs convolution on this embedding. Perhaps the biggest practical difference is that D-Conv has fewer parameters (does not perform the embedding) and does not reduce dimensionality from input to output.
Technical Concerns
------------------
My biggest concerns are that the D-Conv has a strong reliance on nearest neighbors. This means the D-Conv has not much accomodation for "outlier" points that are far from others. The X-Conv operator has some nice properties in this regard (it changes coordinate systems so neighbor locations are centered around the current point), but I don't see this in the D-Conv operator, as in Eq. 2, where the coordinate locations are fed directly into the weighted sum after global rescaling to (0,1). I would imagine that data with outliers (whose values are unlike most others) would dramatically hurt performance, as the weights of D-Conv would need to be shared equally by outliers and inliers.
Experimental Concerns
---------------------
Is there a good reason to not try to compare on publicly available datasets like those used in the PointCNN paper (focusing only on the non-temporal versions of the model)? Using proprietary datasets makes following up on this work a bit hard, would be nice to have some reproducible experiment.
It's not clear to me that the experiments here consider realistic scenarios. Why would I predict mobile app traffic using only the past 30 minutes of data? Why predict air quality using only the last 12 hours? Certainly there are time-of-day, day-of-week, and seasonal effects that are all important. At a minimum, I'd think that for the mobile traffic case you could at least look at consuming the last 48 hr of data and predicting the next 30-90 minutes. I suspect that would make even simpler models do much better.
Further, I think the experiments are missing some key simple baselines (or I misunderstand something). For example, rather than the complicated CNN/LSTM architectures, why not try to directly see how much value there is in "neighbors" in this 2d space? At each point, you can make predictions using only the K nearest neighbors' data, with K swept from 1 to 100 or something. I would expect with these features, using just a simple MLP or RNN would do quite well. I'd like to see a stronger qualitative case made for why we expect the complicated DConv weighting operator here to do better than this baselines.
Overall, the results tables appear promising (for app traffic forecasting in Table 1, the proposed CloudLSTM achieves 3.66 MAE compared to 4.95 for PointCNN and 4.8 for an MLP). However, it's not clear why and I'd like to understand why. Is it that the other approaches are overfitting?
Minor Concerns
--------------
I would suggest avoiding calling the method "\mathcal{D}-Conv", and instead use just "DConv", since this is easier to type into search engines and easier to search for in a PDF document
Related: Point clouds could be represented as graphs, and then use graph embeddings as feature representations


The information bottleneck principle in (Tishby et al., 2000)  is an elegant and useful approach to representation learning. In this paper, we investigate the problem of representation learning in the context of reinforcement learning using the information bottleneck framework, aiming at improving the sample efficiency of the learning algorithms. We analytically derive the optimal conditional distribution of the representation, and provide a variational lower bound. Then, we maximize this lower bound with the Stein variational (SV) gradient method (originally developed in (Liu & Wang, 2016; Liu et al., 2017)). We incorporate this framework in the advantageous actor critic algorithm (A2C) (Mnih et al., 2016)  and the proximal policy optimization algorithm (PPO) (Schulman et al., 2017). Our experimental results show that our framework can improve the sample efficiency of vanilla A2C and PPO significantly. Finally, we study the information bottleneck (IB) perspective in deep RL with the algorithm called mutual information neural estimation(MINE) (Belghazi et al., 2018). We experimentally verify that the information extraction-compression process also exists in deep RL and our framework is capable of accelerating this process. We also analyze the relationship between MINE and our method, through this relationship, we theoretically derive an algorithm to optimize our IB framework without constructing the lower bound.

This paper proposes a representation learning algorithm for RL based on the Information Bottleneck (IB) principle. This formulation leads to the observed state X being mapped to a latent variable Z ~ P(Z | X), in such a way that the standard loss function in actor-critic RL methods is augmented with a term minimizing the mutual information between X and Z (which can be seen as a form of regularization). This results in a loss that is difficult to optimize directly in the general case: the authors thus propose to approximate it through a variational bound, using Stein variational gradient descent (SVGD) for optimization, which is based on sampling multiple Z_i’s for a given state X, so as to compute an approximate gradient for the parameters of the function mapping X to Z. Experiments show that when augmenting the A2C algorithm with this technique, (1) the mutual information I(X, Z) decreases more quickly (better « compression » of the information), and (2) better sample efficiency is observed on 5 Atari games (with also encouraging results with PPO on 3 Atari games).
In spite of the interesting theoretical contributions, I have to recommend rejection as the current empirical evaluation of the proposed approach is extremely limited, making it difficult to assess its benefits over more straightforward algorithms.
On the positive side, the authors derive a sensible approach to IB representation learning in RL, and provide solutions to the optimization challenges it leads to. I did not have time to check all the maths in the Appendix (I only went through the derivations in A.1 and A.2), but they seem to make sense overall (though it is unclear to me if the new algorithm proposed in A.5 is a practical one, so I am not taking it into account in this evaluation).
The key negative point is definitely the weak empirical evaluation. The main results are from a limited sample of 5 Atari games, when the full Atari benchmark has 10x more games and is known to exhibit high variance among games when comparing RL algorithms (the additional results from the Appendix on 3 additional games also show situations where the proposed method does not seem to help much, confirming that larger scale experiments are needed for a proper evaluation). In addition it seems like each algorithm is run only once (instead of using multiple seeds) and only over ~200K timesteps, which is three orders of magnitude lower than results typically reported on Atari. Another issue is that there is no comparison to other representation learning techniques (like those mentioned in the related work section, or the recent "Unsupervised State Representation Learning in Atari"), nor to a natural and more straightforward variant of the proposed method where Z would simply be sampled from a (learned) Gaussian distribution Z ~ N(mu(X), var(X)), which at first sight seems like an easier-to-optimize objective (using the reparameterization trick)… I may be wrong, but then this should probably be explained in the paper (I realize that the proposed approach is more general, but then it should be shown how this extra flexibility can lead to improved results). Finally, the impact on runtime performance is not analyzed: how much slower do A2C / PPO become when optimizing the mutual information term with SVGD? Overall it is really unclear that better RL results can be obtained through this technique.
Another important issue is that I found the paper rather difficult to follow, due to some inconsistent / unclear notations or equations. Here are the main ones I noted:
•	The discount factor is not accounted for in the derivation of the objectives in eq. 1-2 (I know this is often the case in practice, but the reason for dropping it should at least be mentioned)
•	The jump from V^pi(X) to V^pi(Z) at the end of Section 3 is explained too succintly. It suggests that V^pi(Z) must be constant (equal to V^pi(X)) over all values of Z that may be sampled from X, which as far as I can tell is not the case in the rest of the paper. It is also unclear whether pi depends on Z or X. And the notation J(Z) makes it look like J does not depend on X, but it seems like it does because even if pi depends only on Z, by its definition V^pi(Z) actually still depends on X (this also leads to weird equations like eq. 33 where X does not appear in the right-hand side). Overall this is rather confusing.
•	The « for every X » at the top of p. 4 does not make sense to me, due to the term I(X, Z) in eq. 3 where X is a random variable and thus does not take a specific value.
•	The paragraph below eq. 4 is a bit confusing. It looks like it amounts to saying that Y is a Gaussian around V(Z)?
•	Eq. 5 suggests that R depends on Z, but shouldn’t it depend on X? If it depended on Z, then wouldn’t it influence the optimization since by modifying P(Z|X) we can control the distribution on Z and thus the distribution on R?
•	In eq. 10 it is unclear whether L1 and L2 contain the expectation, also L2 is defined as a function of both theta and phi but seems to depend only on theta
•	Below eq. 15 it is said that « P is the distribution of Z » but P does not appear in eq. 15
•	In eq. 16 should the phi on the left hand side be phi* as in eq. 15?
•	Below eq. 16 it is said « Notice that C has been omitted », but it is unclear whether it was not included to alleviate notations or because it disappears naturally in the mathematical derivation of eq. 16
•	The motivation for introducing zeta below eq. 17 is unclear, especially since it seems to play an important role considering that zeta = 0.005 << 1 is used in the experiments (with no explanation as to how this specific value was selected)
•	\hat{L}(Z_i, theta, phi) (between eq. 17 and 18) does not seem to be defined
•	What is the motivation for using the Gaussian U(Z) as described in Section 5? In particular I find it weird that it depends on X_i, while U(Z) is supposed to replace the marginal P(Z) and not the conditional P(Z | X)
Minor points:
•	In the definition of Y_t = R_t = sum_i=0^n-2 … above eq. 4, I think the sum should be up to n-1 for an n-step return
•	Eq. 4 uses R_t on the right hand term instead of Y_t which looks weird
•	Theorem 1 states « Assume that for any epsilon > 0, … »: « for any » should probably be « there exists », since if the inequality was true for any epsilon, it would imply both mutual informations are equal (also the formulation of the theorem does not make it clear that the last inequality is the main result)
•	Footnote 1 p. 4: I(X, Y) should be I(X, Z)
•	In eq. 15 the phi below the argmax should be in bold
•	Please add a reference that the reader can refer to in order to understand where eq. 20 is coming from
•	« Apparently » is used in a couple of places but should probably be replaced with another word
•	When U is uniform, how do you choose its support?
•	After « median of pairwise distances between the particles », I think i=1 should be j=1
•	It is unclear to me what « A2C with noise is »: it is said that the same phi(X, epsilon) is used « as A2C with our framework », but is it the same phi as A2C with uniform SVIB or A2C with Gaussian SVIB? And whichever it is, how does it differ from the one it is equal to?
•	« we add 21 to all four curves in order to make exponential moving average »: I do not understand that sentence
•	« we set the number of samples as 26 for the sake of computation efficiency »: I fail to see how going from 32 to 26 is going to make a major difference in computational efficiency
Update: thank you for your response, but in the absence of a revised version addressing my concerns (as well as those from the other reviewers), I cannot increase my rating


The problem of distributed representation learning is one in which multiple sources of information X1, . . . , XK are processed separately so as to extract useful information about some statistically correlated ground truth Y . We investigate this problem from informationtheoretic grounds. For both discrete memoryless (DM) and memoryless vector Gaussian models, we establish fundamental limits of learning in terms of optimal tradeoffs between relevance and complexity. We also develop a variational bound on the optimal tradeoff that generalizes the evidence lower bound (ELBO) to the distributed setting. Furthermore, we provide a variational inference type algorithm that allows to compute this bound and in which the mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Experimental results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms which we develop in this paper.

This paper studies a distributed representation problem where multiple features X_1,...,X_K are processed (or encoded) separately to estimate (or decode) some quantity of interest Y.
The log loss is considered throughout, which amounts to measuring the mutual information between Y and \hat Y, defined as the "accuracy" of the estimation method. The average rate (measured in number of bits per sample) of the encoded feature is defined as the "complexity" of the representation method.
The author derived the fundamental trade-off between the accuracy and the complexity for any representation-estimation (or encoding-decoding) method.
The author also derived a variational representation of the optimal accuracy-complexity region, which also expresses the optimal encoder and decoder map as the solution of the optimization problem.
Finally, the author considered the case where the joint distribution of P_{X_1,,,,X_K,Y} is unknown, and encoder and decoder are parameterized by neural networks, parameters of which are tuned using data.
I incline to reject the paper, for the following reasons.
1. The accuracy-complexity trade-off studied in the paper is more of a rate-distortion type of information-theoretic problem, where the joint distribution P_{X_1,,,,X_K,Y} is assumed to be known. Its connection to the learning problem, where the joint distribution P_{X_1,,,,X_K,Y} is unknown, is unclear. Even if the precise accuracy-complexity region is obtained, it says little about the sample complexity needed by a learning algorithm to achieve this region.
2. Deriving the optimal encoder-decoder mapping from the variational representation of the accuracy-complexity region also requires the joint distribution, which violates the basic assumption of the learning problem.
3. The author did consider the case where the joint distribution is unknown, and the encoder-decoder pair is learned from data. However, this learning problem is somewhat artificial: each encoder only encodes one of the features, but in order to encode optimally, it has to know the entire joint distribution, hence need to access all the features during training. This discrepancy of seeing different components of the data set during training and inference is not well-motivated.
The author mentioned "multi-view learning" at the beginning of the paper. It would be good if the author can elaborate more on this problem in Sec 4 of Experiment Results, and discuss with more detail on how the proposed method solves this problem and how it is different from the existing results, both in terms of the algorithm and the performance.
=================================================
Feedback to authors' reply
I got a better understanding on how the proposed learning algorithms works after reading the authors' reply.
I guess the idea for the case where the joint distribution is unknown is that, for encoding, different nodes uses its own training data (without accessing other nodes' data) to optimize the encoder separately; while for decoding, the master node trains the decoder uses data available to all nodes to estimate the joint distribution.
In this way, the encoders and the decoder jointly optimizes a variational lower bound of the optimal rate region.
If this is the case, I think the proposed method may have some value in practice.
But now the question is how good the variational lower bound is compared to the optimal region, and how well can this variational lower bound be approximated by neural networks and how efficient can the training be done. Without theoretical analysis on these questions, one may only use experiments to assess the performance. From Table 2, it looks like the improvement of the proposed method on the existing method is quite marginal.
In summary, I would like to thank the authors' valuable reply. I encourage the authors to study the gap between the variational lower bound and the optimal region, and maybe do more experiments to find a good use case of the proposed method.


Learning group representation is a commonly concerned issue in tasks where the basic unit is a group, set or sequence. Previously, the computer vision community tries to tackle it by aggregating the elements in a group based on an indicator either defined by human such as the quality or saliency of an element, or generated by a black box such as the attention score or output of a RNN. This article provides a more essential and explicable view. We claim the most significant indicator to show whether the group representation can be benefited from an element is not the quality, or an inexplicable score, but the discrimiability. Our key insight is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t. the element space can be distilled by a light-weight auxiliary distillation network (DDNet). This procedure is denoted as discriminability distillation learning (DDL). We show the proposed DDL can be flexibly plugged into many group based recognition tasks without influencing the training procedure of the original tasks. Comprehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efficiency, and it pushes forward the state-of-the-art results on these tasks by an impressive margin.

This paper proposes a discriminability indicator using embedded class centroids on a proxy set, and show the discriminability distribution w.r.t. the element space distilled by a light-weight auxiliary distillation network, which is called discriminability distillation learning (DDL).
This methodology was tested on a selection of datasets addressing set-to-set face recognition and action recognition. I have the following concerns, which made me to suggest rejecting this paper.
1) The methodology explained in Section 3 should be improved. In the way it is presented, it is not clear and reproducible. In detail,
1.1)  It is better to describe the pipeline together with Figure 1.
1.2) Contribution of the top part of Figure 1 in proposed pipeline is not clear. More descriptive caption should be added.
1.3) In section 1, where you ask reader to observe the feature embedding of elements which lie close to the centroid… Which section or figure are you referring to?
1.4) In subsection 2.3, clearly discuss how proposed approach is different than traditional distillation methods. Also, explain your modifications and discuss why new structure is better than default ResNet-18.
1.5) Subsections 3.2-3.4 do not contain enough explanation to support equation of proposed method.
1.6) It is not clear why authors use cosine distance, because it is simple?
1.7) The quality scores generated by DDL is very hard to interpret, which to me the one of the biggest problem of the proposed methodology.
2) The related work section should include studies more related to the motivation behind DDL (“….to select represent samples from the whole set efficiently for group understanding.”). For instance, attention models, saliency detection, key-frame detection methods, outlier detection methods, quality aware networks, etc. should be discussed and compared with DDL. Instead, authors more focused on face recognition and action recognition tasks themselves. Therefore, many related works are missing. Especially, Section 2.3., should not be a part of related work but should be integrated either to Section 1 (Introduction) or Section 3.
3) Experimental analysis:
3.1 ) It is not possible to figure out if the proposed method is really performing better than SOA. In other words, the experimental analysis should be extended. One thing authors should consider is including more datasets, especially more challenging datasets i.e., the ones not already saturated. Some examples can be: iLIDS-VID, PRID2011, YouTube Face, IJB-A benchmark.
3.2) Is there any reason to mainly focus on face recognition and action recognition?
3.3) “For face recognition, DDL can easily improve the performance by concentrating the discriminative information and for video action recognition, DDL can further accelerate the pipeline by eliminating the frames with insufficient information.” This sounds like DDL behaves differently depends on the task, but perhaps the conclusion driven by the authors cannot be fully correct given that the number of used datasets are limited to make such a general statement. I suggest authors to re-write this sentence.
3.4) The experimental analysis should include comparisons with methodologies such as quality learning via attention mechanism, etc. to better understand the necessity and effectiveness of proposed DDL. However, only for one dataset such a comparison was performed. It is strange why for other datasets the same comparison was not done.
3.5) Experimental analyses so much focuses on to justifying the performance improvements of DDL is independent to the base model selected (e.g. mainly Table 3,  but also Table 4, 5 and 6 include). However, to me this is a supporting experiment and it is more important to show the real necessity of the proposed method by comparing it with SOTA methods.
3.6) I do not find using Youtube Face benchmark suitable as it is already saturated, i.e. many existing models perform around 97%, which does not allow to show whether DDL can significantly contribute to the task or not. From the corresponding results, it looks like DDL does not provide any significant improvement.
3.7) In Table 3, it is hard to observe the contribution of DDL in terms of the performance. The improvements are in the level of 1%. More discussion is needed about that table.
3.8) “The discriminability distillation learning is more practical to untrimmed video action recognition since there are more diversity videos chip with ambiguous content and visual blur problem”  This means that DDL should not be used in some tasks, it is not generalizable, task-specific? Please elaborate it more clearly.
3.9) Table 2 is hard to interpret, improve the discussions regarding it.
3.10) Table 1 and table 2 were not referred in the text. What does DATA refer to in Table 1, why they are different?, if they are really different i.e. it is not me who misinterpret, then is not it unfair to compare the results?
4) Other suggestions:
4.1) Support your statements with references and appropriate experimental analysis.
4.2) In Section 4, only YTF dataset is mentioned but authors also use IQIYI-VID-FACE challenge dataset.
4.4) Section 4.2. is not clear at all, and should be re-written.
4.5) Some references to the tables are wrong or some are missing, pls. check them all. E.g., in section 4.2., paragraph two, table 3 should be table 2.
4.6) There is no reference to the any figures in the text.
4.7) Paper contains several typos and grammatical mistakes. Also, most of the points were not explained well.


Imperfect information games are challenging benchmarks for artificial intelligent systems. To reason and plan under uncertainty is a key towards general AI. Traditionally, large amounts of simulations are used in imperfect information games, and they sometimes perform sub-optimally due to large state and action spaces. In this work, we propose a simulation reweighing mechanism using neural networks. It performs backwards verification to public previous actions and assign proper belief weights to the simulations from the information set of the current observation, using an incomplete state solver network (ISSN). We use simulation reweighing in the playing phase of the game contract bridge, and show that it outperforms previous state-of-the-art Monte Carlo simulation based methods, and achieves better play per decision.

This paper deals imperfect information games, and builds a Bayesian method to model the unknown part of the current state, making use of the past moves (which constrain the game, here Contract Bridge).
This new Bayesian method is compared to Monte Carlo style techniques, which are much more computationally expensive (they draw random samples of the unknown part of the information and then solve the perfect-information version of the game, for each simulated possible full-state).
The work also introduces a Neural Network (NN) to estimate the best moves in the perfect-information version of the game (instead of making the full tree search of the optimal moves to play).
The final model proposed (ISSN+SSN) uses a NN combined with Bayesian computation, using the NN at each time step of the past to update the belief about the current missing information.
Overall the paper is written clearly: as a non-expert in RL, I was able to follow rather easily what is done.
However, the results are not convincingly good.  Maybe it is just the interpretation/contextualization that is insufficient.  I have 3 important remarks:
1. It is stated in the abstract "that it [the new method] outperforms previous state-of-the-art Monte Carlo simulation based methods, and achieves better play per decision."  However, the costs in the 'reweighing DDS' section (table 2) are lower (better) than those of the 'reweighing SSN' section.  The results show improvement upon using some reweighing, as ISSN+sthg is better than without ISSN (although by a very small relative decrease in cost).
1.a. I understand that the NN-based methods (last three lines of table 2) are incomparably faster than the DDS-based approaches (baseline). And the total rate of tricks loss per initial state remains low (6% seems small as an absolute value).  But, they are almost double of the DDS-based loss rates !  It is not clear how this can be considered 'outperforming' or 'better play per decision'.  If you can explain in which sense those are good results for the SSN vs DDS, please do so. If they are not, please do acknowledge it, and eventually contextualize (maybe a 2-fold increase of loss rate is okay given the large speedup obtained?).
1.b. in the same way, it is not clear how ISSN outperforms its no-memory counterparts, given the loss decreases are essentially negligible. Maybe one needs to increase the value of T to make ISSN's success more obvious?
2. In addition, why not compare this Bayesian method with other Bayesian methods (quoted at the very end of section 3)?  Here the paper focuses on comparison with 'deterministic'  methods, i.e. methods which sample complete states (simulations) to then solve the complete-information version of the game (via exhaustive tree search).  Those are what I would call brute-force methods.
However, although simulations may be done (expensively) for contract bridge, in some other cases this kind of sampling may become prohibitively expensive, so that only Bayesian methods are left.  If this kind of argument is the justification for your work, please make it more explicitly.  Otherwise please correct me and explain the context (the role of Bayesian methods, what has been done and what hasn't been) more clearly.
If you are to situate the work within the Bayesian-based approaches, the question remains: is ISSN+SSN better than other Bayesian-inspired, "information-completing" methods ?
To summarize, the paper convincingly shows that Bayesian-NN methods can compete with expensive brute force methods, but it would be very nice to see how the method introduced compares with other recent Bayesian approaches. (Or if no such comparison can be done, explain why).
3. In addition, here it seems your baseline is based on GIB, which was winning tournaments ~20 years ago: why not compare with Wbridge5 or JackBridge ?  I think you need to at least explain your choice.
Because of these weak points in terms of experimental results, I lean to reject the paper.  However, depending on the authors answers and clarifications on my 3 important remarks above, I am ready to change my rating.
Also, I have a couple of more or less minor remarks to improve the paper:
The definition of the cost is not explicitly given: "We set up the evaluation metric by tricks loss per deal. The ground truth play is DDS result for the original deal and all simulation deals. We compare all the following algorithmS with the ground truth to get costs of the policy."
You should rephrase this to make the definition of 'cost' very explicit. Is it simply the average rate of lost tricks (per given set of 13 cards in the agent hands) ?
This is quite crucial and make the reading of results a bit complicated (especially since results fo not match the conclusion announced in the abstract).
"DDS) 3 computes the maximum tricks  each side can get if all the plays are optimal"
In this place and a couple others, you should explicitly recall whether you mean 'assuming perfect information', or not.  Sometimes it can get confusing, and a bit of repetition won't hurt.  I think I understood correctly that DDS solves (perfectly) the perfect information game, but at times I thought other methods also made use of the full information (?)
Figure 4c is nicely explained and this part really illustrates well the idea of the method, I like it. Although, the notation AQ3 was not obvious for me at first, and I think it is worth improving this figure, making use of the right-hand-side space, to make it an autonomously explanatory figure.
'position': the term is not defined. I would guess it means the current state of the game (the agent's hand and the cards played in the past or during the current trick). This should be said explicitly.
Also "hands" seem to refer to the 4 hands (1 for each player) (is that correct?)
There are wrong singular/plurals ('s'/no 's') in several places. This is simple to correct and should be corrected.
in section 7, ablation studies. This is a very nice study, but you should precise how much you augment the data here (or recall by how much, if you say it earlier).
This passage is unclear and should be rephrased for clarity:
"We run DDS on all 2.4 million positions, to get the
corresponding results for each available action. This serves as ground truth of the current position.
For each position, we run 50 simulations to compute the baselines and serve as training target of the
pretrained network. Each call to DDS takes about 5 milliseconds."
This passage is unclear and should be rephrased for clarity:
"For simplicity, in this work we just use discarding information."
This passage is slightly unclear and should be rephrased for clarity:
"For each simulation, the moves with the optimal number of
tricks are marked with optimal moves. We sum up the optimal moves counter in these k simulations.
This results in a counter for each legal move and we treat this as the training target. The process is
described in Figure 1.""


Delusional bias is a fundamental source of error in approximate Q-learning. To date, the only techniques that explicitly address delusion require comprehensive search using tabular value estimates. In this paper, we develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "consistent" with the underlying greedy policy class. We introduce a simple penalization scheme that encourages Q-labels used across training batches to remain (jointly) consistent with the expressible policy class. We also propose a search framework that allows multiple Q-approximators to be generated and tracked, thus mitigating the effect of premature (implicit) policy commitments. Experimental results demonstrate that these methods can improve the performance of Q-learning in a variety of Atari games, sometimes dramatically.

A recent paper by Lu et al introduced delusional bias in Q-learning, an error due to the max in the Bellman backup not being consistent with the policy representation implied by the greedy operator applied to the approximated value function. That work proposed a consistent algorithm for small and finite state spaces, which essentially enumerates over realizable policies. This paper proposes an algorithm for overcoming delusional bias in large state spaces. The idea is to add to the Q-learning objective a smooth penalty term that induces approximate consistency, and search over possible Q-function approximators. Several heuristic methods are proposed for this search, and results are demonstrated in Atari domains.
I found the topic of the paper very interesting - delusional bias is an intriguing aspect of Q learning, and the approach of Lu et al is severely limited to discrete and small state spaces. Thus, tackling the large state space problem is worthy and definitely not trivial.
The authors’ proposed solution of combining a smooth penalty for approximate consistency and search over regressors makes sense. The implementation of the search (Sec 3.4) is not trivial, and builds on a number of heuristics, but given the difficulty of the problem, I expect that the first proposed solution will not be straightforward.
I am, however, concerned with the evaluation of the method and its practicality, as reflected by the following issues:
1. The method has many hyper parameters. The most salient one, \lambda, the penalty coefficient, is changed between 0.25 to 2 on the consistency penalty experiment, and between 1 to 1000 in the full ConQUR experiments. I did not understand the order of magnitude change between the experiments, and more importantly, how can one know a reasonable \lambda, and an annealing schedule for it in advance.
2. I do not understand the statistical significance of the results. For example, with the constant \lambda=0.5, the authors report beating the baseline in 11 out of 19 games. That’s probably not statistically significant enough to claim improvement. Also, only one run is performed for each game; adding more runs might make the results clearer.
3. The claim that with the best \lambda for each game, the method outperforms the baseline in 16 out 19 games seems more significant, but testing an optimal hyper parameter for each game is not fair. Statistically speaking, *even if the parameter \lambda was set to a constant zero* for the 5 runs that the method is tested on, and the best performing run was taken for evaluation against the baseline, that would have given a strong advantage to the proposed method over the baseline….
4. For the full ConQUR, there are many more hyper parameters, which I did not understand the intuition how to choose. Again, I do not understand how the results establish any statistically significant claim. For example, what does: “CONQUR wins by at least a 10% margin in 20 games, while 22 games see improvements of 1–10% and 8 games show little effect (plus/minus 1%) and 7 games show a decline of greater than 1% (most are 1–6% with the exception of Centipede at -12% and IceHockey at -86%)” mean? How can I understand from this that ConQUR is really better? Establishing a clearer evaluation metric, and using well-accepted statistical tests would greatly help the paper. At the minimum, add error bars to the figures!
5. While evaluating on Atari shows applicability to large state spaces, it is hard to understand from it whether the (claimed) advantage of the method is due to the delusional bias effect, or some other factor (like implicit regularization due to the penalty term in the loss). In addition, it is hard to understand the different approximations in the method. For example, how does the proposed consistency penalty approximate the true consistency? These could all be evaluated on the simple MDP example of Lu et al. I strongly advise the authors to add such an evaluation, which is easy to implement, and will show exactly how the approximations in the approach deal with delusional bias. It will also be easier to demonstrate the effects of the different hyper parameters in a toy domain.


PRIVACY-PRESERVING REPRESENTATION LEARNING
Deep learning and latest machine learning technology heralded an era of success in data analysis. Accompanied by the ever increasing performance, reaching super-human performance in many areas, is the requirement of amassing more and more data to train these models. Often ignored or underestimated, the big data curation is associated with the risk of privacy leakages. The proposed approach seeks to mitigate these privacy issues. In order to sanitize data from sensitive content, we propose to learn a privacy-preserving data representation by disentangling into public and private part, with the public part being shareable without privacy infringement. The proposed approach deals with the setting where the private features are not explicit, and is estimated though the course of learning. This is particularly appealing, when the notion of sensitive attribute is "fuzzy". We showcase feasibility in terms of classification of facial attributes and identity on the CelebA dataset. The results suggest that private component can be removed in the cases where the the downstream task is known a priori (i.e., "supervised"), and the case where it is not known a priori (i.e., "weakly-supervised").

This paper investigates the use of variational auto-encoders (VAEs) and disentanglement to create high quality data representations that hide sensitive attributes. The authors consider two settings: (1) a supervised setting where both the sensitive labels and the downstream machine learning task labels are available to the data holder, and (2) a weakly supervised setting where only sensitive attributes are available. For both settings, the authors propose creating data representations that have 2 components: component 1 captures any information in the data bout the sensitive attributes, and component 2 captures everything else (especially the information useful for a downstream machine learning task). The goal is to ensure that these two components are disentangled.  This is done by training classifiers on each component: 2 classifiers that try to reconstruct the sensitive attributes from each component separately (and in the supervised setting, 2 other classifiers that guess the downstream ML task from each component separately). The overall loss captures all components and ensures that one cannot reconstruct the sensitive attributes from component 2.
Overall, the paper addresses an interesting and timely problem that is of great relevance to this community. However, despite being generally clear, the paper has many grammatical errors and typos. In its current shape, the paper cannot be published.
More importantly, the paper has a number of issues that need to be improved:
1. The introduction makes a number of claims that are incorrect. For instance, the introduction claims that under differential privacy (and refers to Abadi et al.), machine learning models are learned from anonymized data and that complex training procedures run the risk of exhausting the budget before convergence. This is not true because under the classical setting of differential privacy, the models are trained on clean data BUT the process of learning is anonymized (see DP-SGD from Abadi et al.). More importantly, recent research and results in this space indicate that one can train high quality machine learning models with strong differential privacy guarantees (check McMahan et al. ICLR18: Learning Differentially Private Recurrent Language Models). Further, the authors claim that Federated Learning does not allow for the use of the trained model in a central setting. This is also not true because federated learning is an approach to train models on massively decentralized data -- in a way that is orchestrated by a service provider. The learned model can be used in a centralized or decentralized service. Moreover, the communication cost of training models under federated learning may be lower than communicating the data (several hundreds if not thousands of high resolution images) to a central service provider, so the claims about communication are not always correct. Finally, the authors claim that federated learning and differential privacy are useful when the private attributes are known a priori and these approaches fail to provide privacy protections when the private information contained in the dataset isn't identified. This is wrong. Both approaches do not require the knowledge of private information. They, however, require the knowledge of the downstream machine learning task -- something that may not always defined a priori.
2. The privacy guarantees provided by the authors' approach are rather weak and unclear. (1) They assume that a trusted data holder already access to the dataset and wants to release private representations. In the supervised setting, why not just revealing the learned model? In the weakly supervised setting, where will the downstream machine labels come from? Are these representations released only for unsupervised learning tasks? This should be clarified -- and the experiments at the end should reflect this. (2) The privacy guarantees are with respect to (a) a computationally and statistically bounded adversary (i.e. there may very well be stronger adversaries with access to side information that can perfectly reconstruct the sensitive attributes),  and (b) pre-defined sensitive attributes (i.e. there may be other sensitive attributes that one can learn from the published representations that are not captured by this framework).
3. The paper makes no attempt to properly survey the literature on learning representations under censorship and fairness constraints. For example, they do not reference and compare against: (a) Censoring Representations with an Adversary (https://arxiv.org/abs/1511.05897), (b) Learning Adversarially Fair and Transferable Representations (https://arxiv.org/abs/1802.06309), (c) Context-Aware Generative Adversarial Privacy (https://arxiv.org/abs/1710.09549), (d) Learning Generative Adversarial RePresentations (GAP) under Fairness and Censoring Constraints (https://arxiv.org/abs/1910.00411), and many others. Without a clear (empirical) comparison to these works, the benefits of the proposed approach are unclear.
4. The proposed method for disentanglement does not scale to non-binary sensitive attributes (because of the blowup in the number of classifier pairs that would need to be trained to ensure disentanglement). Thus, this approach may be limited to cases with a few sensitive attributes.
5. The authors are encouraged to show the learned representations (so that one could verify with the naked eye that the representations are indeed disentangling the sensitive attributes).


Deep learning applied to the reconstruction of 3D shapes has seen growing interest. A popular approach to 3D reconstruction and generation in recent years has been the CNN decoder-encoder model often applied in voxel space. However this often scales very poorly with the resolution limiting the effectiveness of these models. Several sophisticated alternatives for decoding to 3D shapes have been proposed typically relying on alternative deep learning architectures. We show however in this work that standard benchmarks in 3D reconstruction can be tackled with a surprisingly simple approach: a linear decoder obtained by principal component analysis on the signed distance transform (SDF) of the surface. This approach allows easily scaling to larger resolutions. We show in multiple experiments it is competitive with state of the art methods and also allows the decoder to be fine-tuned on the target task using a loss designed for SDF transforms, obtaining further gains.

This paper studies the problem of learning the feature representation for predicting the 3D shape of objects, from a single image or a point cloud. The proposed approach performs PCA on the SDF field. And then the transformed feature map is learned and used as input to task-specific decoders for 3D shape prediction. The authors claims that this approach trains faster and is easier to scale, while showing competitive performance compared to state-of-the-art methods.
I am leaning towards Weak Reject. The paper is generally easy to read, but with some details missing. And I found the discussion of the results to be insufficient. I think it can be an above-threshold paper if questions are addressed during rebuttal.
Being able to easily scale to higher resolutions is claimed to be one of the main advantages, but I am not convinced that this is useful under this setting. If I understand this correctly, the number of eigenvectors k is fixed, and projecting the SDF field to this space would remove the higher frequency components of the shape. So wouldn't the number of eigenvectors be the bottleneck in representational precision, not the resolution of the output space?
What is the chosen k (number of eigenvectors)? It says k was "chosen to capture 99.5% of the variance within the dataset", but I could not find how exactly it was chosen and what value of k was used (I apologize if I missed).
Also, I think the PCA is category-specific (page 4, section 4.1). Is k dependent on the category or is it the same across all categories? Some of the other methods (if not all) used for comparison are not category-specific, so if this is true, I think the comparison may not be entirely fair and it should be made clearer.
I think the writing could be polished as well, some minor typos:
Page 2:  3rd and 4th paragraphs: continous
Page 2: anlaysis, enlightning
Page3: under eigenSDF: reprsentation
Page 4: section 4.1: refered, signficant
Page 5: “seciton”
Page 7: tranform
Page 3, Section 3.2: Is N the number of training examples and M the resolution?
Figure 2 not referred in the main text.


Despite the success of deep learning in domains such as image, voice, and graphs, there has been little progress in deep representation learning for domains without a known structure between features. For instance, a tabular dataset of different demographic and clinical factors where the feature interactions are not given as a prior. In this paper, we propose Group-Connected Multilayer Perceptron (GMLP) networks to enable deep representation learning in these domains. GMLP is based on the idea of learning expressive feature combinations (groups) and exploiting them to reduce the network complexity by defining local group-wise operations. During the training phase, GMLP learns a sparse feature grouping matrix using temperature annealing softmax with an added entropy loss term to encourage the sparsity. Furthermore, an architecture is suggested which resembles binary trees, where group-wise operations are followed by pooling operations to combine information; reducing the number of groups as the network grows in depth. To evaluate the proposed method, we conducted experiments on five different real-world datasets covering various application areas. Additionally, we provide visualizations on MNIST and synthesized data. According to the results, GMLP is able to successfully learn and exploit expressive feature combinations and achieve state-of-the-art classification performance on different datasets.

This paper addresses the problem of learning expressive feature combinations in order to improve learning for domains where there is no known structure between features. These settings would normally lead to the use of fully-connected MLP networks, which unfortunately have problems with efficient training and generalization after a few layers of depth. The main idea is to use grouping at first, in combination with smaller fully-connected layers for each group, as well as pooling pairs of groups together as the layers go on. Results are shown as comparisons on 5 real-world datasets, and intuitive visualizations on two other datasets. Related work covered MLPs, regularization techniques, sparse networks, random forest models, and other feature grouping. The paper is well written and easy to read. This work did a good job with giving implementation details as well as performing hyperparameter searches and giving the baselines a good effort.
My current decision is a weak reject, for a well-written paper, but some concerns as follows:
-The results do not show much improvement (i.e., < 0.3% improvement for 3 of the datasets, and < 1% for another one), aside from CIFAR-10. Considering that the premise of the paper is that MLP’s are not good enough when dealing with data in which the relationships between features are unknown, it seems like these are definitely not good datasets on which to demonstrate this notion of “there has been little progress in deep reinforcement learning for domains without a known structure between features.”
-The MNIST visualization of group-select felt informative, but the XOR example for grouping visualizations seemed too easy. It would’ve been good to see visualizations or intuitions regarding grouping for harder datasets, in order to be convinced of the need for more expressive feature representations than standard MLP’s.
-I’m not an expert on causality, but it seems like citations from that area are required for this problem statement of dealing with features where the connections between them are unknown but potentially very important.
Less major:
-It would have been nice to include related work on other ways to encourage inter-feature interactions, such as perhaps taking the outer product of the input with itself.
-It seems like different sizes per group would be a more realistic expectation, and that perhaps this should be worked into the algorithm. Similarly, pooling only 2 groups together (from pre-specified positions) seems like it would be limiting as well. It also seems like the algorithm should account for being able to use a high-level feature from one layer as part of multiple groups in the future (i.e. reuse). Even if any of these options don’t make a difference, it would be good to check/evaluate.
Minor:
-Equation 8 did not fully make sense to me.
-Why were “random horizontal flips” used as preprocessing for the permutation-invariant CIFAR-10 dataset? This shouldn’t make a difference at all if the pixels become randomly shuffled anyway.


Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.

Update: I thank the authors for their response. I believe the paper has been improved by the additional baselines, number of seeds, clarifications to related work and qualitative analysis of the results. I have increased my score to 3 since I still have some concerns. I strongly believe the baselines should be tuned just as much as the proposed approach on the tasks used for evaluation. The baselines were not evaluated on the same environments in the original papers, so there is not much reason to believe those parameters are optimal for other tasks. Moreover, the current draft still lacks comparisons against stronger exploration methods such as Pseudo-Counts (Ostrovski et al. 2017, Bellemare et al. 2016) or Random Network Distillation (Burda et al 2018).
Summary:
This paper proposes the use of a generative model to estimate a Bayesian uncertainty of the agent’s belief of the environment dynamics. They use draws from the generative model to approximate the posterior of the transition dynamics function. They use the uncertainty in the output of the dynamics model as intrinsic reward.
Main Comments:
I vote for rejecting this paper because I believe the experimental section has some design flaws, the choice of tasks used for evaluation is questionable, relevant baselines are missing, the intrinsic reward formulation requires more motivation, and overall the empirical results are not convincing (at least not for the scope that the paper sets out for in the introduction).
While the authors motivate the use of the proposed intrinsic reward for learning to solve tasks in sparse reward environments, the experiments do not include Moreover, some of the tasks used for evaluation do not have very sparse reward (e.g. acrobot but potentially others too). Without understanding how this intrinsic reward helps to solve certain tasks, it is difficult to assess its effectiveness. While state coverage is important, the end goal is solving tasks and it would be useful to understand how this intrinsic reward affects learning when extrinsic reward is also used. Some types of intrinsic motivation can actually hurt performance when used in combination with extrinsic reward on certain tasks.
I am not sure why the authors chose to not compare against VIME  (https://arxiv.org/pdf/1605.09674.pdf) and NoisyNetworks (https://arxiv.org/pdf/1706.10295.pdf) which are quite powerful exploration methods and also quite strongly related to their our method (e.g. more so than ICM).
Other Questions / Comments:
1. You mention that you use the same hyperparameters for all models. How did you select the HPs to be used? I am concerned this leads to an unfair comparison given that different models may work better for different sets of HPs. A better approach would be to do HP searches for each model and select the best set for each.
2. Using only 3 seeds does not seem to be enough for robust conclusions. Some  of your results are rather close
3. How did you derive equation (1)? Please provide more explanations, at least in the  appendix.
4. Why is Figure 3 missing the other baselines: ICM & Disagreement? Please include for completeness
5. Please include the variance across the seeds in Figure 4 (b).
6. How is the percentage of the explored maze computed for Figure 5? Is that across the entire training or within one episode? What is the learned behavior of the agents? I believe a heatmap with state visitation would be useful to better understand how the learned behaviors differ within an episode. e.g. Within an episode, do the agents learn to go as far as possible from the initial location and then explore that “less explored” area or do they quasi-uniformly visit the states they’ve already seen during previous episodes?
7. In Figure 6 (b), there doesn’t seem to  be a significant difference between your model and the MAX one. What happens if you train them for longer, does MAX achieve the same or even more exploration performance as  your model? I’m concerned this small difference may be due to poor tuning of HPs for the baselines rather than algorithmic differences?
8. For the robotic hand experiments, can you  provide some intuition about what the number of explored rotations means and how they relate to a good policy? What is the number of rotations needed to solve certain tasks? What kinds of rotations do they explore -- are some of them more useful than others for manipulating certain objects? This would add context and help readers understand what those numbers mean in practice in terms of behavior and relevance to learning good / optimal policies.