It is fundamental and challenging to train robust and accurate Deep Neural Networks (DNNs) when semantically abnormal examples exist. Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused on and how much more should they be emphasised to achieve robust learning? In this work, we study this question and propose gradient rescaling (GR) to solve it. GR modifies the magnitude of logit vector's gradient to emphasise on relatively easier training data points when noise becomes more severe, which functions as explicit emphasis regularisation to improve the generalisation performance of DNNs. Apart from regularisation, we connect GR to examples weighting and designing robust loss functions. We empirically demonstrate that GR is highly anomaly-robust and outperforms the state-of-the-art by a large margin, e.g., increasing 7% on CIFAR-100 with 40% noisy labels. It is also significantly superior to standard regularisers in both clean and abnormal settings. Furthermore, we present comprehensive ablation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios.

Summary
This paper presents Gradient Rescaling (GR) for robust learning to combat label noise. They propose to treat each data sample with different significance scores: some samples are important to learning, and some examples are insignificant (or even detrimental) to learning. So they desire to weight each samples according to their significance. They propose the notion of emphasis focus (When learning, whether we should put emphasis on learning “hard” examples or “easy” examples) and emphasis spread (the variance of these significance weights). The authors propose that this “difficulty” of samples are proportional to their network output logit values.
The authors examine the analytical forms of the gradients of popular loss functions such as Categorical Cross Entropy, Mean Absolute Error and Generalized Cross Entropy. They find that the formulas for the gradient are of similar family with varying hyperparameters. Authors claim that tweaking these hyperparameters result in tuning the emphasis focus and spread.
The authors conduct Experiments on CIFAR10, CIFAR100 with simulated symmetric noise. Also, they conduct experiments on real-world noisy datasets: Clothing 1M dataset and MARS video dataset. The authors claim that the performance of GR exceeds various baselines.
Significance/Novelty/Clarity
Significance: Low-Medium. The performance increase exhibited in the experiments are a bit underwhelming (when considering the fact that benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are missing).
Novelty: Medium. The paper is interesting in the sense that the authors integrated (and allegedly generalized) the gradient formulas for several losses into one family, and tried to integrate and tweak their postulation of  “Emphasis focus” and “emphasis spread” into the framework. However, the theoretical ground and convincing reasoning for their claim seems a bit lackluster.
Clarity: Low. The overall flow of the paper is a bit fuzzy - exhibiting a stream-of-consciousness style flow.
Pros and Cons in Detail
Pros:
1.The authors try to unify the analytical forms of the gradients of various loss functions into a single family equipped with hyperparameters that control emphasis focus and spread.
2.Conducted experiments show that GR achieved increased performance when compared to the baselines.
Cons:
My major concern is about tuning newly introduced hyperparameters in practical settings. How can we guarantee to have intact validation set?  Can we get any improvement via GR even with corrupted validation set for tuning hyperparameters?
1. The arguments of the authors are grounded in the premise that “difficult” samples will exhibit small logit values, and “easy” samples high logit values.
2. No justifications (both theoretical and experimental) are provided on the claim that controlling emphasis focus/spread will result in more robust learning.
3. This algorithm introduces 2 additional hyperparameters that are correlated with each other. This introduces additional labor.
4. By changing the loss function, the outputs of the network might lose its interpretation as a probability distribution.
5. No confidence intervals are shown except for the CIFAR-100 experiment.
6. Experiments are only conducted on vision tasks.
7. The baseline menagerie also changes when the authors change the target dataset.
8. Additional benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are required.
Questions
1. Is it always the case that “difficult” samples exhibit small logit values, and “easy” samples high logit values?
2. If not, GR’s emphasis manipulation might result in neglecting samples containing valuable information.
3. Can GR be used simultaneously with other noise-robust learning methods to further boost the performance?
4. Technically, GR aims to rescale the gradients of the logits. How will it interact with optimizers  other than SGD such as Adam?
5. Does GR still work well on small datasets(#points < 5000)?
Misc. Comments
Page 3-> inside L1 norm, no differentiation sign in the denominator.
Around eq 2 and 4: missing derivative symbol w.r.t. z


Recent works have empirically shown that there exist adversarial examples that can be hidden from neural network interpretability (namely, making network interpretation maps visually similar), and interpretability is itself susceptible to adversarial attacks. In this paper, we theoretically show that with the correct measurement of interpretation, it is actually difficult to hide adversarial examples, as confirmed by experiments on MNIST, CIFAR-10 and Restricted ImageNet. Spurred by that, we develop a novel defensive scheme built only on robust interpretation (without resorting to adversarial loss minimization). We show that our defense achieves high classification robustness, outperforming state-of-the-art adversarial training methods against attacks of large perturbation while attaining high interpretation robustness under various settings of adversarial attacks.

The present work considers adversarial attacks that also yield similar outputs for "interpretability methods", which are methods that output some vector corresponding to a given classification (usually the vector is e.g. an image or a similar object). It also shows that by regularizing nearby inputs to have similar interpetations (instead of similar classifications), robustness can be achieved similar to adversarial training.
I did not understand the motivation of the paper. Why is it important for adversarial attacks to yield similar interpretations? A human would need to assess the interpretations to detect the attack, but it would already be trivial for an attack to be detected given human oversight (just check whether the classification of the image matches the human-assigned label). It also wasn't clear how this was related to the other observation that regularizing based on interpretability yields robustness; these seem like two fairly separate results.
Finally, I found the claim that "interpretability alone helps robustness" to be misleading and not substantiated by the paper. The purported justification is that regularizing nearby inputs to have the same interpretation yields robustness. But a better summary of this observation is that "robustness of interpretability implies robustness of classification", which is not surprising, and is in fact a trivial corollary of the fact that the metric on interpretations dominates the classification error metric (an observation which is made in the paper).
More minor, but I found it hard to follow the writing in the paper (this is related to the motivation being unclear). This is exacerbated by the paper being longer than unusual (10 pages instead of 8).


Recent advances in the sparse neural network literature have made it possible to prune many large feed forward and convolutional networks with only a small quantity of data. Yet, these same techniques often falter when applied to the problem of recovering sparse recurrent networks. These failures are quantitative: when pruned with recent techniques, RNNs typically obtain worse performance than they do under a simple random pruning scheme. The failures are also qualitative: the distribution of active weights in a pruned LSTM or GRU network tend to be concentrated in specific neurons and gates, and not well dispersed across the entire architecture. We seek to rectify both the quantitative and qualitative issues with recurrent network pruning by introducing a new recurrent pruning objective derived from the spectrum of the recurrent Jacobian. Our objective is data efficient (requiring only 64 data points to prune the network), easy to implement, and produces 95 % sparse GRUs that significantly improve on existing baselines. We evaluate on sequential MNIST, Billion Words, and Wikitext.

Notes:
  -RNN network pruning has proven to be challenging using the techniques often used with other network types.
  -One issue is that the performance of an LSTM/GRU can hinge on a few activated gates and can lead to more concentration of influence than would be seen in a feedforward network without parameter sharing between layers.
  -New objective uses 64 points to prune the network (I assume this is just the size of the minibatch).
  -Result is a 95% sparse GRU cell.
  -New idea is based on keeping weights that propagate information through many time steps.
  -Encourages "singular values of the temporal jacobian with respect to network weights to be non-degenerate" (I suppose this means that the gradient flowing through time will contain multiple directions of variation)
  -Introduction does a good job of introducing the key ideas.
  -J_t is a temporal jacobian of size N x N (N is number units) at time step t.
  -Chi is the spectral norm of this temporal jacobian.  I'm a bit confused by this, because my understanding is that the spectral norm is the largest singular value, but equation 3 looks like a sum over singular values, making it more like a frobenius norm?
  -This jacobian isn't tractable, so paper approximates it using a first-order taylor expansion.  So basically the pruning just amounts to taking parameters with the largest gradient?
  -Section 2.4 is confusing and seems to come out of nowhere.  Is this suggesting that the technique isn't just pruning but adding a new normalization scheme?  On second reading, this is a normalization scheme effecting which parameters to prune.  The motivation for why the gradients are normalized like this is still confusing.  If you're willing to make a linear assumption, it seems like it's enough to consider the gradient on the parameter multiplied by the magnitude of the parameter to see the overall effect of removing it?
  -The results look good, but sequential mnist is a bit of a toy task.  I'd also like to see a more fine-grained analysis showing the tradeoff between the number of units removed and the performance.
  -The paper claims that L2 pruning requires more data, but it's unclear if this really matters since the whole dataset was used to train both methods initially.
  -On Table 2 the results of the technique don't seem that much better than "Random".
  Review: This paper presented a fast pruning algorithm for RNNs, which uses the norm of the gradient as a guide to pruning.  I'm borderline on this paper.  The idea of using the gradient is good, but the explanation of some aspects like the normalization is confusing and felt random.  Additionally the results, while better than some other pruning techniques on RNNs, don't seem to be that much better than random.


The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold, and SP-LIME aggregates the local models to form a global interpretation. Extending this line of research, we propose a simple yet effective method, NormLIME, for aggregating local models into global and class-specific interpretations. A human user study strongly favored class-specific interpretations created by Norm-LIME to other feature importance metrics. Numerical experiments employing Keep And Retrain (KAR) based feature ablation across various baselines (Random, Gradient-based, LIME, SHAP) confirms NormLIME's effectiveness for recognizing important features.

This paper proposes a novel scheme to aggregate local explanations
from LIME to generate global and/or class-level interpretations in the
form of feature importances. The technical novelty lies in the
normalization of the feature weights across local explanations where
the proposed scheme utilizes a well-motivated L0 normalization in
place of the existing L2 normalization in Ribeiro et al., 2016. The
sparsity induced by the L0 normalization leads to more "easy-to-read"
global (or class level) "explanations" since the number of features
with non-zero importances is reduced.
The simple idea of generating sparse aggregations of local
explanations to obtain global (or class-level) interpretations is well
motivated and seems fairly intuitive (and I consider simplicity a
strength) and the extremely favourable empirical results, I am leaning
towards a reject for the current version of the paper. The main
reasons for this decision are as follows:
- The proposed scheme is presented as closely tied to LIME, which
  severely limits the scope of the proposed solution. However, I can
  imagine that the high level idea of sparse aggregation of local
  explanations can easily generalize to many local explanation
  schemes.
- The empirical evaluation needs to be improved (or better clarified)
  in my opinion. There are many potential baselines that are also not
  considered (see details below). Moreover, the evaluation is
  completely limited to a single data set (which is good as a strawman
  but not sufficient to make significant claims).
Clarifications re: baselines and empirical evaluations:
- No comparison to Ibrahim, et al., 2019 -- you can cluster in
  unsupervised setting but you can also aggregate within class in
  supervised setting. Generating class-level interpretations is an
  interesting idea but its novelty is somewhat unclear.
- In terms of explanations, there are contrastive explanations [1]
  which are not really discussed in this paper. This scheme could be
  more useful for generating class level interpretations.
- It is not clear why the negative correlations were removed in the
  user study. In certain situations, such as when distinguishing
  between digits 3 & 8, the absence of weights (potentially
  corresponding to negative weights) can be crucial explanations.
- It is not clear why in the SmoothGrad and VarGrad baselines, the
  class level interpretations are generated with just 10 images per
  class. Are the class level interpretations of LIME and NormLIME also
  generated from just 10 images per class. If that is the case, 10
  seems like a very small number and can potentially create noisy
  explanations. If LIME and NormLIME used more than 10 images to
  generate class level interpretations, the comparison does not seem
  fair. Please explain this discrepancy.
- For the evaluation in Sec. 5, it is not clear how the features are
  removed. Are they removed based on the global explanations (feature
  importances) of each baseline? If this is the case, are the global
  explanations of each baseline generated using the same amount of
  local explanations? Please clarify.
- The KAR analysis results in Fig. 4(b) need to be further
  investigated. First, it is not clear if this difference between the
  different baselines something that happens in multiple datasets or
  just this one. Moreover, it seems natural that something like
  SmoothGrad and VarGrad would be able to capture the feature
  redundancies if the redundancies were the actual cause for this
  marked difference between the relative performances in Fig. 4(a) and
  4(b). This could definitely use a better discussion.
[1] Dhurandhar, Amit, et al. "Explanations based on the missing:
Towards contrastive explanations with pertinent negatives." Advances
in Neural Information Processing Systems. 2018.


Adversarial examples are malicious inputs crafted to cause a model to misclassify them. In their most common instantiation, "perturbation-based" adversarial examples introduce changes to the input that leave its true label unchanged, yet result in a different model prediction. Conversely, "invariance-based" adversarial examples insert changes to the input that leave the model's prediction unaffected despite the underlying input's label having changed. So far, the relationship between these two notions of adversarial examples has not been studied, we close this gap. We demonstrate that solely achieving perturbation-based robustness is insufficient for complete adversarial robustness. Worse, we find that classifiers trained to be p -norm robust are more vulnerable to invariance-based adversarial examples than their undefended counterparts. We construct theoretical arguments and analytical examples to justify why this is the case. We then illustrate empirically that the consequences of excessive perturbation-robustness can be exploited to craft new attacks. Finally, we show how to attack a provably robust defense -certified on the MNIST test set to have at least 87% accuracy (with respect to the original test labels) under perturbations of ∞ -norm below ε = 0.4 -and reduce its accuracy (under this threat model with respect to an ensemble of human labelers) to 60% with an automated attack, or just 12% with human-crafted adversarial examples.

================ Update after reading the rebuttal and the revised paper =======================================
I have now read the author rebuttal and the revised paper. I had raised two main issues with the paper in my initial review (see below): 1) experiments don't provide enough support for the main claim; 2) the main claim seems to contradict an earlier result by Engstrom et al. (2019).
With respect to the second point, the authors have responded that Engstrom et al. (2019) use relatively small perturbation sizes during adversarial training and that the trade-off between invariance vs. perturbation based adversarial robustness may be different for such small perturbation sizes. This sounds plausible, although I would have much preferred a more direct demonstration of this (for example, by adversarially training MNIST models with different perturbation sizes, then looking at their performance on invariance-based adversarial examples).
With respect to the first point, the authors have pointed to some numbers in Table 1, but my criticism still stands: for example, the perturbation-robust ABS performs better than the undefended CNN model (similarly for the l_inf PGD model, even though it is trained for a different metric). This seems to contradict the authors' claim that excessive perturbation robustness leads to worse performance on invariance-based adversarial examples.
In conclusion, I very much appreciate the authors' rebuttal and I definitely think there are some interesting ideas in this paper. However, given that one of my main concerns has not been addressed and the other one only insufficiently addressed, I'm inclined to keep my score as it is. Going forward, I would encourage the authors to develop the trade-off between invariance vs. perturbation based adversarial sensitivity for different perturbation sizes more thoroughly.
========================================================================================================
This paper argues that the standard notion of l_p-norm bounded adversarial examples does not adequately capture all types of adversarial examples we may care about. In particular, the authors note the existence of adversarial examples caused by the excessive invariance of classifiers to semantically meaningful perturbations. This was noted in earlier works as well, but the current paper purports to establish a link between l_p-ball robustness and invariance-based adversarial examples. It also introduces a method to generate such invariance-based adversarial examples. There are interesting ideas in this paper, however I have some questions and concerns about some of the claims made in the paper that I would like to see addressed. Here are the main issues for me:
1) If I’m not mistaken, the main claim of the paper, namely that l_p-ball robustness worsens the performance on invariance-based adversarial examples does not actually seem to be supported by any result in the paper beyond some of the simplest, toyest examples.  For example, in the MNIST examples in Table 1, I don’t see how l_p-ball robust models are performing worse than the other models on the invariance-based adversarial examples (the results seem mixed at best). Even in some of the toy examples, this claim is not supported: for example, in the adversarial spheres example, the l_p-ball robust classifier would correspond to the max-margin classifier, which is not vulnerable to invariance-based adversarial examples. Even the ad-hoc sub-optimal classifier chosen in this example doesn't show that l_p-ball robust models are more vulnerable to invariance-based adversarial examples, just that the two phenomena are distinct. There’s the single ImageNet example in Figure 6, which is cute and is consistent with the claim, however it’s just a single example. Also please note that this example does not work with the l_inf norm, so it just shows the inadequacy of the l_2 norm, not of l_p norms in general.
2) The main claim of this paper seems to contradict a result in Engstrom et al. (2019) (https://arxiv.org/abs/1906.00945). They also demonstrate the existence of invariance-based adversarial examples in non-robust models (e.g. see their Figure 2), but their results seem to suggest that these types of examples do not arise in robust models (models trained with adversarial training). Please discuss this paper and clarify the seeming discrepancy between the results there and your claims.
Other issues:
3) It is really hard to follow the invariance-based adversarial example generation process described in section 4.2. Please describe this more clearly, motivating each step of the process (currently the steps seem ad hoc, it is not explained why each step is needed), so that a reader can understand the generation process at a high level without having to consult the appendix.
4) The labels are misaligned in Figure 4.
5) In section 4.3, it is mentioned that “we additionally hand-crafted 50 invariance adversarial examples under the specific norms”. It is not explained at all why these additional examples are needed and it is not described how exactly these are generated (neither in the main text, nor in the appendix as far as I can see).
6) In Table 1, the different models are not described at all (neither in the main text, nor in the appendix). They are just acronyms right now. Please describe what these models are.
7) The idea that there may be trade-offs between different notions of robustness and that l_p-ball robustness is not the be all and end all of all robustness measures have been noted in some previous works: for example, Yin et al. (2019): https://arxiv.org/abs/1906.08988 Although the phenomenon discussed in Yin et al. (2019) is different from the one highlighted in the current paper, the main message seems quite similar. So, please discuss this connection in your paper.


Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.

The paper extends recently proposed BatchRenormalization (BRN) technique which uses exponential moving average (EMA) statistics in forward and backward passes of BatchNorm (BN) instead of vanilla batch statistics. Motivation of the work is to stabilize training neural networks on small batch size setup. Authors propose to replace EMA in backward pass by simple moving average (SMA) and show that under some assumptions such replacement reduces variance. Also they consider slightly different way of normalization without centralizing features X, but centralizing convolutional kernels according to Qiao et al. (2019).
Concerns:
Changing batch statistics with moving averages in Eq. (4) and Eq. (16) may introduce bias in stochastic gradients. Authors do not reflect this problem in the work.
Assumption (3) from Theorem 3 does not hold in practice since authors centralize weights not features. Authors do not study the influence of this on the performance of the method.
Authors’ main motivation is to study small batch size experimental setup which is important in segmentation and detection, but they don’t include main competitor (BRN) in these experiments.
Overall, the proposed method has a lack novelty and thorough comparison against BRN. Therefore, I would suggest rejecting the current version.
--------------------------------------------------------
Update after author rebuttal
Thank you for your clarification. Below I provide an updated review for the paper.
The paper proposes the improvement of batch normalization techniques for the case of small batch size. Authors reveal new statistics used in gradient calculation of original BatchNorm and show their instability in case of small batch size. Thereafter they improve the method by reducing the number of statistics in backward pass resulting in more stability and performance increase. The authors provide a good experimental study on the influence of individual parts of the method. However, I still have concerns about the strictnesses of theorems assumptions and interpretability of their results in practice (i.e., o(1) biases in Theorem 1, o(1) assumptions in Theorem 2).
Overall, almost all of my concerns were justified. Therefore I increase my score to 6.


Motivated by applications to machine learning and imaging science, we study a class of online and stochastic optimization problems with loss functions that are not Lipschitz continuous; in particular, the loss functions encountered by the optimizer could exhibit gradient singularities or be singular themselves. Drawing on tools and techniques from Riemannian geometry, we examine a Riemann-Lipschitz (RL) continuity condition which is tailored to the singularity landscape of the problem's loss functions. In this way, we are able to tackle cases beyond the Lipschitz framework provided by a global norm, and we derive optimal regret bounds and last iterate convergence results through the use of regularized learning methods (such as online mirror descent). These results are subsequently validated in a class of stochastic Poisson inverse problems that arise in imaging science.

The paper establishes optimal regret bounds of the order O(\sqrt{T}) for Follow The Regularised Leader (FTRL) and Online Mirror Descent (OMD) for convex loss functions and potentials (a.k.a. Riemannian regularizers) that are, respectively, Lipschitz continuous and strongly convex with respect to a given Riemannian metric. These conditions naturally generalize the classical conditions typically considered in the literature, which are defined with respect to a global norm and, as such, are not well-suited to problems where the loss functions and its gradient present singularities at the boundary of the feasibility region. The authors suggest a principled way to choose both the Riemannian metric and the potential function based on the singularity landscape of the gradient of the loss function. Via standard online-to-batch conversion, the authors also address the offline setting and give O(1/\sqrt{T}) error bounds for ergodic averages in convex problems and for last iterates in non-convex problems satisfying a weak secant inequality. The authors include numerical experiments involving a Poisson inverse problem.
The paper is well-written, with a very clean narrative highlighting the main ideas and results. To the best of my knowledge, the literature review is complete and rightly highlights the fact that most results in the literature on Riemannian mirror descent methods have so far primarily addressed offline deterministic problems with exact oracle gradients. The contribution of this work lies not only in the focus on online and noisy setting but also on establishing natural results upon natural generalizations of well-known conditions in standard (non-Riemannian) settings. The techniques used are extensions of the classical theory and follow quite naturally, with the exception of the non-trivial primal-dual inequality (20).
QUESTIONS/SUGGESTIONS:
1) Can the authors be more explicit about how the (OMD) equations are derived from (16). While this is standard, I feel currently there is a bit of a jump in the narrative—which otherwise is very good.
2) The workhorse behind the established results is the primal-dual inequality (20) which relies on the introduction of the Fenchel coupling. Can the authors be more explicit about the use of this inequality, and what makes the Riemann generalization difficult in general? In particular, can the authors comment on the applicability of this inequality (or similar) to the smooth setting?
3) Typo: sometimes the notation $\mathcal{U}$ seems to be used instead of the notation $\mathcal{X}$. See, for instance, equation (7) in Definition 1 and the definition of the set $\mathcal{Z}$ in Remark 2.
AFTER REBUTTAL: I thank the authors for addressing my questions.


Recent research shows that for training with 2 loss, convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel (CNN-GP) (Novak et al., 2019) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel. Here we show how to significantly enhance the performance of these kernels using two ideas. (1) Modifying the kernel using a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression. This idea is inspired by Global Average Pooling (GAP), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation. (2) Representing the input image using a pre-processing technique proposed by Coates et al. (2011), which uses a single convolutional layer composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012) , and outperforms the best previous classifier that is not a trained neural network (Mairal, 2016). Similar improvements are obtained for Fashion-MNIST.

This paper shows that there is a one-to-one correspondence between pixel-shift based data augmentation and average pooling operations in CNN-NNGP/NTK based ridge regression. Interestingly, the authors show that standard average pooling + flatten can lead to a better performance than simple global average pooling. This paper further shows that using the data pre-processing step proposed in (Coates et al., 2011) can boost performance of CNN-NNGP/NTK based ridge regression by ~7% which allowed the authors to achieve classification accuracy in high 80s which is AFAIK SOTA on CIFAR-10 when not using learned representations.
My current assessment of the paper is “weak accept”. There are two main reasons why I am on the verge of recommending rejection of this paper: (1) I believe that the experiment evaluation is not done entirely correctly leading to inflation of the reported results (my guesstimate is by ~0.5-2%)---please see my “Major comments”. If this is not fixed, I am very likely to downgrade my score. (2) While the observation of the relationship between pixel-shifts and average pooling is very nice (which is why my current score is “weak accept”), it seems that most of the improvement comes from application of the pre-processing step of Coates et al. (2011) (seems like a ~7% improvement!). Given the large computational cost of CNN-NNGP/NTK (authors say about 1000 GPU hours), I wonder whether a simpler algorithm like some of the newer variants of boosting combined with the Coates et al. algorithm wouldn’t also perform at around 87-88% like CNN-NNGP/NTK (given the baseline 85-86% accuracy of the Coates et al. (2011) algorithm reported by the authors).
Major comments:
- Can you please clarify why you decided to give a new name (Box Blur) to standard average pooling? Why not just use the existing name?
- I believe that the way you report results in all the tables (i.e., tables 1-6) and the text based upon them is flawed. The right approach would be to select the hyperparameters “c” and “d” on a validation set, and then report the performance with these hyperparameters on the test set. While the experiments are somewhat rescued by the fact that you report results for (almost) all the possible hyperparameter settings (which allows us to see samples from the population distribution of the generalisation error), type-setting the best results in boldface and thus implying that these are valid estimates of the generalisation error is not appropriate since you are effectively selecting the best hyper-parameters on the test set! Unfortunately, I cannot accept these results to be published “as-is”. While re-running the experiments with hyperparameter selection on validation set is already a somewhat imperfect solution, I am not sure I can see a better way forward. However, I do understand that this could be prohibitively expensive in which case I would like to ask you to suggest an alternative solution please (of course, other reviewers are welcome to chime in as well)?!
- While most of the paper is about Local Average Pooling (LAP) and the equivalence between averaging and pixel shifts, the experimental results seem to show that most of the improvement comes from the use of Coates et al.’s preprocessing step. Could you please run the experiments in tables 3 and 4 with c=0 and c=32 to see what the effect of the preprocessing is without LAP?
- In sect.6.3, you say “Our experiment illustrates that even with a fixed last FC layer, using GAP could improve the performance of CNN, and challenges the conjecture that GAP reduces the number of parameters in the last fully-connected layer and thus avoids overfitting.” I am not sure I see why fixing the last FC layer should provide more convincing evidence than training it? I do not know the conjecture to which you refer but from your description, the overfitting without GAP should occur because the FC layer has more parameters than with GAP?! If this is true, then the overfitting would happen in the last layer (due to the large number of parameters) which you have (at least partially) prevented by not training it?! Can you clarify and also report the results of this experiment with all the layers trained please?
- In Appendix D, you say that you have used lambda = 10^{-5} for all configurations. How have you selected this particular value please? Do you have a sense of how far from optimal this value is for all the different configurations (or at least for NTK vs NNGP models---in my experience, the optimal setting between the two can differ quite a bit)?
Minor comments:
- In the abstract and throughout the paper, you claim that the cost of kernel regression is quadratic. AFAIK without any approximations, the cost is cubic (or O(n^{2.67}) to be more precise). Please clarify.
- In par.1 on p.1, you say “convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) **goes to infinity**” (emphasis mine) and cite the Jacot et al. (2018) paper. AFAIK this paper only works with infinite networks but does not actually prove that **deep** networks of finite width (in each layer) converge to the NTK limit; IMHO you should cite the Allen-Zhu et al. (2018) and Du et al. (2018) papers from your references for that result. Based on p.2 (end of par.2 in sect.2), you seem to be aware of this distinction but cite Arora et al. (2019) instead of these two; I would suggest either citing Allen-Zhu et al. and Du et al. only, or citing all three as the Arora et al. paper came out later than the first versions of the other two paper which AFAIK already contained all the necessary derivations (even if the words “Neural Tangent Kernel” were not spelled out there).
- Also in par.1 on p.1, you say that Arora et al. (2019) was the first to provide an algorithm to compute the CNTK kernel which is a bit of a stretch given that both Garriga-Alonso et al. (2019) and Novak et al. (2019) have implemented the CNTK kernel in their experiments. AFAIK the claim in (Arora et al., 2019) is that they provided first **efficient** implementation of the CNTK-GAP kernel which should be made clearer in the next revision of your paper.
- On p.2, you say “These kernels correspond to neural networks where only the last layer is trained.” In reality, the correspondence is not exact for finite networks because the induced kernel will not be exactly equal to the one at the limit.
- Bottom of p.2, “Global Average Pooling (GAP) is proposed” -> “... was proposed”.
- Top of p.3, “..., and GAP is more robust” -> “..., and that GAP is more robust”.
- On p.3 in the “Padding Schemes” paragraph, do you mean to assume that the input image has only a single channel (not necessary later)?
- On p.4, I am slightly confused by your definition of the “augmented kernel”. Specifically, it does not seem K^G (x , x’) = K^G (x’, x) holds in general. Can you please clarify? If there’s no symmetry, I do not think it necessary to use a different name, but perhaps a clarifying note would be beneficial to the reader?!
- On p.5, fig.1 is too small when printed and one needs to use the computer screen to see what is depicted; given the amount of white space around, can you please try to make the images larger (you can perhaps also only include 2 or 4 images instead of 16 which will give you additional space)?
- On p.5, you say that for small “c”, circular padding will not create unrealistic images. Looking at fig.1b, it seems like the images are not as unrealistic as in fig.1a but human eye can still tell they are not realistic (potentially even more so with other images than the one selected for this figure). I am not sure whether there is a reason to assume this issue does not affect CNNs too?! Further, I am not convinced the motivation is correct in the first place given that the optimal “c” for CIFAR-10 is 12 which will presumably create clearly unrealistic images; perhaps it would be best to omit this motivation?!
- On p.6, you claim “Another advantage of LAP is that it **does not incur any extra computational cost**” (emphasis mine) while at the next line you say that there is a constant additional computational cost. Perhaps say that the extra computational cost is relatively small?
- It might be nice to swap tables 3 and 4 so at least the results for NNGP are next to each other. Even better would be the current table 3 was closer to table 1 to achieve the same effect for NTK.
- I am not sure I fully understand the description in sect.6.3: isn’t the number of channels on the input irrelevant after computation of the kernel in the first layer? In other words, why have you opted to use only 2,048 patches in your experiments and not 32,000 or 256,000 as used by Recht et al. (2019)? Do you have an estimate of how different could the performance of NNGP/NTK be with the larger number of features? Do you know what is the performance of Coates et al.’s algorithm with only 2,048 features? Relatedly, do you know how AlexNet would perform if its PCA data augmentation was replaced by the Coates et al.’s feature extractor?


This paper proposes a representational model for image pair such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1). The model couples the following two components. (1) The vector representations of local contents of images. (2) The matrix representations of local pixel displacements caused by the relative motions between the agent and the objects in the 3D scene. When the image frame undergoes changes due to local pixel displacements, the vectors are multiplied by the matrices that represent the local displacements. Our experiments show that our model can learn to infer local motions. Moreover, the model can learn Gabor-like filter pairs of quadrature phases.

The authors propose a model for learning local pixel motions between pairs of frames using local image representations and relative pixel displacements between agents and objects.  The model learned is compared to the ability of the primary visual cortex where adjacent simple cells share quadrature relationships and capture local motion.
"The representation theory underlies much of modern mathematics and holds the key to the quantum
theory (Zee, 2016)."
Can the relevance of this claim be elaborated on?
"Figure 1 illustrates the scheme of representation."
Please provide more detail here on what is happening in the figure.  The caption and reference here are not informative to what the figure is representing.
"We obtain the training data by collecting static images for (It) and simulate the
displacement field ...  We refer to this method as self-supervised learning"
This is not self-supervised learning.  In self-supervised learning the training label/signal is generated by the system.  In this case artificial data is being generated as the displacement between images is sampled.
Since the motion between images is artificially generated what guarantees are there that the model is learning to capture realistic motion behavior?  Why not use adjacent video frames?
"Note that those methods train deep and complicated neural networks with large scale datasets to
predict optical flows in supervised manners, while our model can be treated as a simple one-layer
network, accompanied by weight matrices representing motions."
Is there a comparison on execution times of the different approaches?
"by obtaining the pre-trained models and testing on V1Deform testing data"
Is this a fair comparison if the proposed approach was trained on V1Deform training data and the comparison methods were not.  A more appropriate comparison would be to apply all the methods to infer the displacement fields between video frames which is also a more natural application.  This can be controlled to contain small motions if needed.  Why nt use the MUG dataset here?
"Displacements at image border are leaved out" -> left out
Sections 5.4, 5.5 and 5.6 show only qualitative results with no comparison methods.  Can the authors provide reasons that other methods could not be used for evaluation?
I am not sure I understand the motivation for the approach.  Why do we need this over other methods that can better capture larger motions.  This needs to be more clear from the introduction.  Why do we care if the approach captures aspects of V1 for the tasks presented?
The work is sensible and the approach is clear but I found the evaluation and motivation lacking in key areas that I mention above.  The authors should revise and make it clear to the reader why we should care about this problem.  Aligning with V1 is interesting but it does not come into play in the applications of the approach or the analysis so I am not sure why I should care.  The evaluation also needs to be much more convincing before I could recommend acceptance.


Although the state-of-the-art object detection methods are successful in detecting and classifying objects by leveraging deep convolutional neural networks (CNNs), these methods overlook the semantic context which implies the probabilities that different classes of objects occur jointly. In this work, we propose a context-aware CNN (or conCNN for short) that for the first time effectively enforces the semantics context constraints in the CNN-based object detector by leveraging the popular conditional random field (CRF) model in CNN. In particular, conCNN features a context-aware module that naturally models the mean-field inference method for CRF using a stack of common CNN operations. It can be seamlessly plugged into any existing region-based object detection paradigm. Our experiments using COCO datasets showcase that conCNN improves the average precision (AP) of object detection by 2 percentage points, while only introducing negligible extra training overheads.

This paper introduces a context-aware neural network (conCNN) that integrates context semantics into account for object detection. The proposed approach achieves this by embedding a context-aware module into the Faster R-CNN detection framework. The context-aware module simulates the learning process of Conditional Random Fields (CRF) model using a stack of common CNN operations. Specifically, this paper employs the mean-field approach of [1]. Experiments are performed on COCO dataset.
The  paper reads well. The idea of combining the strengths of both CNNs and CRF-based graphical models in a detection framework is interesting. Though the main idea is borrowed from the mean-field approach of [1], it has been successfully reformulated as a stack of common CNN layers. However, my main concern is that the experimental results (Page 8 Table 2) does not support the merits of the proposed approach. When comparing with the existing Faster R-CNN + Relation work of [2], the improvement provided by the proposed approach is negligible. In fact the proposed approach achieves inferior results on large objects (APL), compared to [2]. [1] uses attention to model the geometric relationships among objects in the same image. In addition to geometric relationships, the proposed approach leverages the inherent co-occurence relationships between object classes. However, this additional information does not seem to provide much help by looking at the results on COCO dataset. The paper does present an experiment to justify this additional information in Table 1. However, that small experiment is only on 6 object categories and does not seem to generalize when going towards full COCO dataset with 80 categories (Table 2). Therefore, the reviewer would recommend to thoroughly evaluate the merits of the proposed approach in depth on COCO and on additional datasets (i.e., Visual Genome).
[1] Philipp Krähenbühl, Vladlen Koltun: Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. NIPS 2011.
[2] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, Yichen Wei: Relation Networks for Object Detection. CVPR 2018.


Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the variable sparsity problem (VSP), which describes a phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input, and show that it adversarially affects the model performance. We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness, which we refer to as Sparsity Normalization (SN), that directly targets and resolves the VSP. We further experimentally validate SN on diverse benchmark datasets, to show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks.

This paper provides a novel solution to the variable sparsity problem, where the output of neural networks biased with respect to the number of missing inputs. The authors proposed a sparsity normalization algorithm to process the input vectors to encounter the bias. In experiments, the authors evaluated the proposed sparsity normalization model on multiple datasets: collaborative filtering datasets, electric medical records datasets, single-cell RNA sequence datasets and UCI datasets. Results show that the proposed normalization method improves the prediction performance and the predicted values of the neural network is more uniformly distributed according to the number of missing entries.
The paper describes a clear and specific machine learning problem. Then the authors demonstrate a simple normalization strategy is capable of fixing the issue of biased prediction. The paper has a well-organized structure to convey the motivation. Therefore, my opinion on this paper leans to an acceptation. My questions are mainly on the experiment section:
1) As shown in Table 2, there are various new collaborative filtering methods proposed after 2015, why the authors chose to extend AutoRec (Sedhain et al., 2015) but not other new methods?
2) In the experiments, you compare your model with zero imputation (Please correct me if w/o SN is not zero imputation). However, I think it is a common practice in machine learning that we perform imputation with mean or median values. I'm interested in knowing whether filling with mean/median values work with these datasets.
3) In section 4.5, you mentioned that "SN is effective
even when MCAR assumption is not established". However, I'm still not clear about the reason. I believe many machine learning datasets have NMAR (not missing at random) type of missing data, but not MCAR. So this is an important issue for me.
4) Does your model assume all input values are numerical but not categorical?


While deep learning and deep reinforcement learning systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge, particularly as these algorithms learn individual tasks from scratch. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single task learning are not fully understood. Motivated by the insight that gradient interference causes optimization challenges, we develop a simple and general approach for avoiding interference between gradients from different tasks, by altering the gradients through a technique we refer to as "gradient surgery". We propose a form of gradient surgery that projects the gradient of a task onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task reinforcement learning problems, we find that this approach leads to substantial gains in efficiency and performance. Further, it can be effectively combined with previously-proposed multi-task architectures for enhanced performance in a model-agnostic way.

The paper presents a method to boost multi-task learning performance by editing gradient to remove conflicts between tasks. The main idea is to use cosine similarity to 1) determine if two task gradients conflict and 2) to project one conflicting gradient to the normal plane of the other, thereby removing the conflict at the expense of disturbing the other gradient to some extent. Experiments are presented for classification and other computer vision tasks along with reinforcement learning problems.
Overall, I really liked this paper. The explications are clear, the visualizations provided help the understanding (especially Fig.1), and results are compelling. I definitely value the way the method is straightforwardly presented: the underlying idea is simple yet strong. There are, however, a few elements precluding me to pick a higher rating, which I describe in details here.
First, there are a lot of similarities with the MTL method of Sener and Koltun 2018. In particular, I do not agree with the statement that "[this] work, in contrast to many of these optimization schemes [incl. Sener and Koltun], suggests that the challenge in multi-task learning can be attributed to the problem of gradient trashing, which we address directly by introducing a practical algorithm that de-conflicts gradients from different tasks." MTL has the concept of "common descent direction" and, as Fig.1 of relevant paper suggests, it does "de-conflict" the gradients. Sure, the wording is not the same, but the idea is there nonetheless.
To be clear, this is not to say that PCGrad has no merits. I do find it simpler and more elegant (although the latter is a subjective assessment). But I think the similarities should have been discussed in greater details, and the performance compared with Sener and Koltun on at least one problem.
Second, the experiments make it difficult to see the performances of PCGrad alone. Indeed, it is always combined with another multi-task approach/algorithm (MTAN, WPL, SAC+PA, etc.). Providing these results is not incorrect in itself, but it makes it difficult grasp what PCGrad can do alone. Is it worth using only in conjunction with other approaches, or could someone consider using it in a standalone manner? Is PCGrad more a "gradient fine-tuner" than a comprehensive solution for multi-task leearning? The experiments presented, although significant, do not answer these questions.
Third, I am unsure about the value of Theorem 1 and its proof in Sec. 3.2. It assumes too much (e.g., 2 convex tasks) to be of any use in practice. Also, leading to a minimizer of L does not forcibly mean leading to a good solution, depending on the definition of L1 and L2.
There is also one element I'm unsure about. Looking at Algorithm 1, it looks like the order of the tasks may have an effect. Indeed, since it is g_i that is modified at line 8 and not g_j, the last element of B will always remain unaltered (because all other task gradients will have previously been modified to avoid gradient conflict with it). The second to last element of B will be altered, but only due to potential conflicts with the last, and so on, up to the first which can potentially be altered by all others. As an effect, some gradients will always be significantly more altered than others, which can have an impact (at least theoretical) on the learning process. Algorithm 1 thus looks like a greedy approach. Nothing bad in itself (and some ad hoc adjustments, like shuffling B at each update, could very well fix this), but I think this deserves more discussion.
Finally, a more generic comment: the naming of the method should remain the same through the paper. The abstract/intro/Fig.1 refer to the technique as "gradient surgery", while the explanations and experiments talk about PCGrad. Also, in the introduction "plateuas" -> "plateaus".
In summary, I think this is a good paper, presenting a straightforward and useful idea for multi-task learning. However, the related work is not always well described, the experiments lack important comparisons, and the practical effects of PCGrad should be explained in more details instead of focusing on a proof for a convex case.


Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps. However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy. We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity. We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality. For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%. Learnable K-matrices can also simplify hand-engineered pipelines-we replace filter bank feature computation in speech data preprocessing with a kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task. K-matrices can also capture latent structure in models: for a challenging permuted image classification task, a K-matrix based representation of permutations is able to learn the right latent structure and improves accuracy of a downstream convolutional model by over 9%. We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task.

Summary
The authors introduce kaleidoscope matrices (K-matrices) and propose to use them as a substitute for structured matrices arising in ML applications (e.g. circulant matrix used for the convolution operation). The authors prove that K-matrices are expressive enough to capture any structured matrix with near-optimal space and matvec time complexity. The authors demonstrate that learnable K-matrices achieve similar metrics compared to hand-crafted features on speech processing and computer vision tasks, can learn from permuted images, achieve performance close to a CNN trained on unpermuted images and demonstrate the improvement of inference speed of a transformer-based architecture for a machine translation task.

Review
The overall quality of the paper is high. The main contribution of the paper is the introduction of a family of matrices called kaleidoscope matrices (or K-matrices) which can be represented as a product of block-diagonal matrices of a special structure. Because of the special structure, the family allows near-optimal time matvec operations with near-optimal space complexity for structured matrices which are commonly used in deep architectures.
The proposed approach is novel. It gives a new characterization of sparse matrices with optimal space complexity up to a logarithmic term. Moreover, the proposed characterization is able to learn any structured matrix and matvec time complexity of the K-matrix representation is near-optimal matvec time complexity of the structured matrix. Even though in the worst-case complexity is not optimal, the authors argue that for matrices that are commonly used in machine learning architectures (e.g. circulant matrix in a convolution layer) the characterization is optimal. This results in a new differentiable layer based on a K-matrix that can be trained with the rest of an architecture using standard stochastic gradient methods. However, it is worth noting that the reviewer is not an expert in the field, and it is hard for him to compare the proposed approach with previous work.

The paper is generally easy to follow. Even though the introduction of K-matrices requires a lot of definitions, they are presented clearly and Figure 1 helps to understand the concept of K-matrices. The experimental pipeline is also clear.
Given the special structure of the family, the reviewer might guess that having K-matrices can slow down the training, i.e. it might require more epochs to achieve the reported results compared to baselines. Providing training plots might increase the quality of the paper.
The experimental results are convincing. First, the authors show that K-matrices can be used instead of a handcrafted MFSC featurization in an LSTM-based architecture on the TIMIT speech recognition benchmark with only a 0.4% loss of phoneme error rate. Then, the authors evaluate K-matrices on ImageNet dataset. In order to do so, they compare a lightweight ShuffleNet architecture which uses a handcrafted permutation layer to the same architecture but with a learnable K-matrix instead of the permutation layer. The authors demonstrate the 5% improvement of accuracy over the ShuffleNet with 0.46M parameters with only 0.05M additional parameters of the K-matrix and the 1.2% improvement of accuracy over the ShuffleNet with 2.5M parameters with only 0.2M additional parameters of the K-matrix. Next, the authors show that K-matrices can be used to train permutations in image classification domains. In order to demonstrate so, they take the Permuted CIFAR-10 dataset and ResNet-18 architecture, insert a trainable K-matrix at the beginning of the architecture and compare against ResNet-18 with an inserted FC-layer (attempting to learn the permutation as well) and ResNet-18 trained on the original, unpermuted CIFAR-10 dataset. With K-matrix, the authors achieve a 7.9% accuracy improvement over FC+ResNet-18 and only a 2.4% accuracy drop compared to ResNet-18 trained on the original CIFAR-10. Finally, the authors demonstrate that K-matrices can be used instead of the decoder’s linear layers in a Transformer-based architecture on the IWSLT-14 German-English translation benchmark which allows obtaining 30% speedup of the inference using a model with 25% fewer parameters with 1.0 drop of BLEU score.
Overall, the analysis and the empirical evaluations suggest that K-matrices can be a practical tool in modern deep architectures with a variety of potential benefits and tradeoffs between a number of parameters, inference speed and accuracy, and ability to learn complex structures (e.g. permutations).
Improvements
1. Even though K-matrices are aimed at structured matrices, it would be curious either to empirically compare K-matrices to linear transformations in fully-connected networks (i.e. dense matrices) or to provide some theoretical analysis.
2. Section 3.3 argues that K-matrices allow to obtain an improvement of inference speed, however, providing the results of convergence speed (e.g. training plots with a number of epochs) will allow a better understanding of the proposed approach and will improve the quality of the paper.


Hyperparameter optimization is both a practical issue and an interesting theoretical problem in training of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at a negligible additional computational cost, results can be improved by sampling nonlocal paths instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in training, which can be mapped to an effective temperature. The usually independent instances of the model are coupled and allowed to exchange their hyperparameters throughout the training using the well established parallel tempering technique of statistical physics. Each simulation corresponds then to a unique path, or history, in the joint hyperparameter/model-parameter space. We provide empirical tests of our method, in particular for dropout and learning rate optimization. We observed faster training and improved resistance to overfitting and showed a systematic decrease in the absolute validation error, improving over benchmark results.

REVIEW 2
The paper proposes a new paradigm to perform hyperparameter search by proposing a way to jointly optimize over the hyperparameter space and the parameter space as opposed to the traditional way to performing these in separation (with the hyperparameter search invoking the parameter optimization) The paper’s main claim is that this allows the seach to follow non-local paths in the joint space. The main methodology proposed in the paper is inspired by the idea of parallel tempering from physics. The paper proposes to view parameter learning under a certain hyperparameter as running a langevin chain at a particular temperature. This is motivated by considering common hyperparameters as batch size, dropout rate or learning rate as inducing a specific level of noise to the training process, the variance of which is analogous to the inverse temperature in Langevin diffusion.
Under the above analogy, the paper proposes to run an analogue of parallel tempering in the following way. Consider running parallel threads of parameter optimization for various fixed choices of hyperparameter values (each representing a temperature). After running certain number of steps of SGD in each configuration, the paper proposes to exchange hyperparameters within a pair of threads, with the chance of exchange depending upon a criteria motivated by metropolis hastings.
While the paper proposes an interesting idea I have many reservations about the paper and applicability of the idea in practice prompting my score which I outline below.
1. The first and major shortcoming of the paper is an extreme lack of detail and rigour. The paper seems to be written in a very hand wavy fashion. While that is okay in working towards the proposal of the algorithm, which of course is merely claimed to be inspired by parallel tempering. I believe the paper needs to do an important job of specifying exactly how one selects the notion of temperature for each choice of the hyperparameter. The mapping of the choice to the temperature value is essential for the algorithm to succeed and as described in the paper, there is no hope of replicating the algorithm as the authors are extremely vague about how to figure out this mapping. A second instance of this is diffusion curves in the introduction. The paper does not define what a numerical value of diffusion is. It only relies on a hand-waved understanding of diffusion to even understand presented curves. Note that the paper does not have an appendix and there is no link to code either.
2. Lack of intuition - for this being a good procedure in terms of selecting a good path eventually. Note that hyperparameter optimization is an optimization procedure and it is very unclear to me why sampling from the distribution which will look like the product distribution over the replicas is even a good idea. If it is the case this needs to be specified carefully.
3. The experiments performed seem to follow a common trend. A single hyperparameter is selected (learning rate/dropout rate), a small grid of fixed values for the hyperparameter are selected and the procedure is performed. In light of this, what a “good” path through the algorithm represents is a good schedule for adjusting these hyperparameters. This is a well known intuition that for parameters like learning rate one needs to adjust them periodically to achieve good training dynamics. Since the comparison is to fixed learning rates, the improvement therefore looks predictable.
4. Lack of applicability of the algorithm for hyperparameter search in practice - There are a couple of things which make the algorithm restrictive -
This seems to be useful for hyperparameters that can be in a sense equated to parameters that change the training dynamics. The examples considered in the paper - batch size, learning rate, dropout rate. I think the authors should make this distinction and state it upfront because there are many hyperparameters that we optimize over which change the target distribution and not just the noise.
Secondly, large scale hyperparameter search takes the idea of separating hyper parameter and parameter search as then it can run in a parallel asynchronous fashion. The algorithm is highly synchronous way of doing this search. While it can be argued that we should move to this paradigm if the results are significant but unfortunately no comparisons with existing hyperparameter search algorithms are provided.


Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust and efficient measures of uncertainty are crucial. While it is possible to train regression networks to output the parameters of a probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying confidence of its predictions. In this paper, we propose a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target. We accomplish this by placing evidential priors over our original Gaussian likelihood function and training our NN to infer the hyperparameters of our evidential distribution. We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output. Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters. We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.

This paper proposed deep evidential regression, a method for training neural networks to not only estimate the output but also the associated evidence in support of that output. The main idea follows the evidential deep learning work proposed in (Sensoy et al., 2018) extending it from the classification regime to the regression regime, by placing evidential priors over the Gaussian likelihood function and performing the type-II maximum likelihood estimation similar to the empirical Bayes method [1,2]. The authors demonstrated that the both the epistemic and aleatoric uncertainties could be estimated in one forward pass under the proposed framework without resorting to multiple passes and showed favorable uncertainty comparing to existing methods. Robustness against out of distribution and adversarially perturbed data is illustrated as well.
On the technical side, the novelty is incremental. The extension from the classification regime to the regression regime, from the conjugate Dirichet prior to the conjugate Normal-Inverse-Gamma prior, is quite straightforward. Besides, the presentation of the paper could be largely improved. It is not easy to follow the derivation in Section 3. The discussion of concepts and problem definitions look fragmented and incoherent. Even though the presentation largely follows (Sensoy et al., 2018) and uses terms from theory of evidence, the derivation actually is more aligned with the prior network [3] under the Bayesian framework which is missing from the references. It is really confusing that the authors talked about the variational inference when conjugate prior is used, and it is unclear how the variational distributions are used in Section 3.2 or how the "I don't know" loss term relates to the KL-divergence between the variational distribution and the prior in Section 3.3. This term was manually added as additional regularization to "prefer the evidence to shrink to zero for a sample if it cannot be correctly classified" in (Sensoy et al., 2018), and a different regularization was used to encourage distributional uncertainty in [3]. I hope that the authors could spend more efforts clarifying their ideas, especially the derivations in Section 3.2 and 3.3.
On the other hand, there is no referring to the input x in the entire derivation and problem formulation in Section 3. It took me a while to realize that the formulation in (4) actually defines the generation for a particular input, not for all the inputs. That is, the model is trying to model heteroscedastic uncertainty, not the homoscedastic counterpart. It could be better to call out the dependence on the input explicitly.
On the quantitative side, the baseline models considered in Section 4 are mainly concerned with epistemic uncertainty estimation. So it would be good to explicitly discuss which uncertainty estimation was compared with. This work estimates both aleatoric and epistemic uncertainties, so a better comparison is to models that estimate both quantities (Kendall & Gal, 2017)[4] which has been shown to give better output estimation comparing to epistemic uncertainty estimation only (Kendall & Gal, 2017).
Other comments:
- What is the \pi in equation (8)?
- The "I don't know" loss introduced in Section 3.3 used L-p norm. What is the originality of the L-p norm here? In practice, which p value should be used? In the experiments, which p value was used?
- The RMSE results of the depth estimation presented in Table 2 are orders of magnitude smaller than those from existing work, for example Table 2(b) in (Kendall & Gal, 2017). Was a different RMSE computation used in this work?
- From the caption in Table 2, it seems that only 5 samples were used in MC-dropout, which is considerably smaller than those used in existing work (Kendall & Gal, 2017).
[1] D.J.C. MacKay. Hyperparameters: optimize, or integrate out? Maximum Entropy and Bayesian Methods, Springer 1996.
[2] B. Efron. Two modeling strategies for empirical Bayes estimation. Statistical Science, 2014.
[3] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. NeurIPS 2018.
[4] Y. Kwon, J.-H. Won, B.J. Kim, and M.C. Paik. Uncertainty quantification using Bayesian neural networks in classification: application to ischemic stroke lesion segmentation. MIDL 2018.


From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search. In light of this observation, we demonstrate that, against conventional wisdom, Adam can also outperform SGD on vision tasks, as long as the coupling between its learning rate and adaptability is taken into account. In practice, AvaGrad matches the best results, as measured by generalization accuracy, delivered by any existing optimizer (SGD or adaptive) across image classification (CIFAR, ImageNet) and character-level language modelling (Penn Treebank) tasks. This later observation, alongside of AvaGrad's decoupling of hyperparameters, could make it the preferred optimizer for deep learning, replacing both SGD and Adam.

In this paper the authors develop variants of Adam which corrects for the relationship of the gradient and adaptive terms that causes convergence issues, naming them Delayed Adam and AvaGrad. They also provide proofs demonstrating they solve the convergence issues of Adam in O(1/sqrt(T)) time. They also introduce a convex problem where Adam fails to converge to a stationary point.
This paper is clearly written and has reasonable experimental support of its claims. However in terms of novelty, AdaShift was published at ICLR last year (https://openreview.net/forum?id=HkgTkhRcKQ) and seems to include a closely related analysis and update rule of your proposed optimizers. In AdaShift instead of correcting for the correlation between the gradient and eta_t, they correct for the relationship between the gradient and the second moment term v_t. Could you further clarify the differences between the two, both in your approach to deriving the new update rule and the algorithms themselves? Additionally, their Theorem 1 could be compared to yours, but noting the differences for these seems less important. If the optimizers are unique enough, including AdaShift in your experiments would be very useful for demonstrating their differences.
Regarding experiments, while it is true that adaptive methods are supposed to be less “sensitive” to hyperparameter choices, the limits of the feasible ranges for each hyperparameter could vary drastically across problems (especially, as previously demonstrated, across different batch sizes.) Thus, not retuning across experiments seems like it could negatively affect performance for any of the transferred hyperparameter settings. Instead of demonstrating hyperparameter insensitivity by carrying over hyperparameter settings, one could instead retune for each problem and show that a higher percent of hyperparameter combinations result in the same/similar best performance (similar to what is done in Fig. 2, but also showing a (1-dimensional) SGD version which would presumably contain fewer high performing settings.)
Some additional comments:
-The contribution of Theorem 1 is a nice addition to the literature.
-Your tuning of epsilon is great! I believe more papers should include epsilon in their hyperparameter sweeps.
-Scaling epsilon with step size makes sense when considering that Adam is similar to a trust region method, where epsilon is inversely proportional to the trust region radius. However, in section 5 implying that epsilon should be as large as possible in the worst case seems like an odd result given that this would always diminish your second moment term as much as possible, defeating the point of the additional Adam-like adaptivity. Can you comment on why this diminished adaptivity would be desirable in the worst case scenario analyzed?
-The synthetic toy problem is much appreciated, more papers should start with a small, interpretable experiment.
-Was SGD with momentum used? If not, this may not be a fair comparison, as I believe it is much more common to use momentum with SGD. If momentum was used, was the momentum hyperparameter tuned? If not, this may be advantageous to the Adam based methods, as they have more versatile adaptability and thus may not need as much care with their selection of momentum values.
-Was a validation set used for CIFAR? You note in appendix C that there are 50k train and 10k test. You mention validation performance in the main text, so this is just double checking.
-The demonstration in figures 2, 3 of decoupling the step size and epsilon in interesting! Given that the best performing values seem to be on the edges of the ranges tested, I would be curious if the trend continues for more extreme values of alpha and epsilon (one could sparsely search along the predicted trendlines.)
Nits:
-“Vanilla SGD is still prevalent, in spite of the development of seemingly more sophisticated adaptive alternatives...” This could use some citations to back up the claim, because as far as I know it is much more common to use SGD with momentum and is actually rare to use vanilla SGD (the DenseNets and Resnets citations use momentum=0.9.)
-It would be nice to highlight in color the diff from vanilla Adam in the Algorithm sections.
-It is not super clear from the text how in eq 26 you get \sum{E[f(w_t)|Z]} = f(w_1)
-I may be misreading something, but I believe the H in the leftmost term in the last line of eq 33 should be an L.
-In section 5, “for a fixed learning rate (as is typically done in practice, except for discrete decays during training)” seems like an overly broad claim, given that authors commonly use polynomial, linear, exponential, cosine, or other learning rate decay/warmups. Granted for some CIFAR and ImageNet benchmarks there are more common discrete learning rate schedules, but that does not seem to be the overwhelmingly prevalent technique.
Overall, while this area of analyzing Adam and proposing modifications is a popular and crowded subject, I believe this paper may contribute to it if my concerns are addressed. While I currently do not recommend acceptance, I am open to changing my score after considering the author comments!


Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.

* Note: I highlight I did not assess the model design, which is the main contribution of the paper, and  did not know the background of prior work of system design to really assess the novelty of the work, my score is solely based on the experiments.
 I am an expert in machine learning/computer vision, so I could assess the experiments in terms of their validity and relevance from the machine learning/vision perspective, however, I may not be the best person to evaluate the design choices of the system. Therefore I choose the low experience for my background.
* Paper summary: The paper proposes a framework to evaluate machine learning models in a hardware-agnostic way.
To evaluate the models using this framework, the user needs to specify the pre-processing, inference, and post-processing steps and the required software/hardware stack. The authors argue that this is important to consider the  HW/SW stack to allow a fair evaluation and reproducibility. Models are specified using a model specification called manifest.
* The authors assume that SW/HW stack change the results of deep learning models a lot, and this is the main assumption in this work, however, normally in practice HW/SW stack wont change the results.
* I found the experiments either not related to the point of the paper or being very trivial not helping to backing up the arguments of the paper.
* In section 4.1, the authors consider different preprocessing operations and study their impact on the model performance, however, the fact that preprocessing impact the results is trivial in machine learning. In the same section, color layout and data layout, cropping and resizing, where the authors discussed about for instance how changing the data representation from NCHW or NHWC change the results, this is also trivial, because if you change the dimensions, you need to also change the model in a way that handles this change of dimension, therefore, this is clear that the results will change accordingly as well. Such experiments does not back up the main argument of the paper, which argues for fair evaluation between neural models, nor provides informative information to the reader.
On section 4.1, the experiment of type conversion and normalization, again this is mathematically clear that the order would change the results,  let's call imgByte=x, then by substituting given
values for mean and standard evaluation, equation (b) is simplified to (b) = (x-127.5)/((127.5)*(255))
however simplification of (c) results in (x/255-0.5)/0.5 = (x-0.5*255)/(0.5*255)=(x-127.5)/(0.5*255)
the dominator of (b) and (c) are not equal, therefore, this is trivial that the results of these two
the expression would not be the same. The author posed it as a new finding, but this is trivial that mathematically
these two equations would not be equal. Again, this experiment does not add any value to the paper.
In section 4.2, in Figure 9, the authors show a plot of the CPU latency for different batch sizes and instances,
together with GPU throughput for different batch sizes, i.e., images/seconds. The authors show latency for CPU
instances, versus throughput for GPU instance, since these two measures are not shown for both instances, this is
not supported from the text, how actually authors compare this two instance and draw the conclusion that which instance is more efficient since there is no value shown for CPU throughput. Apart from that, I don't see how this section and determining if GPU or CPU instances of  Amazon compute cloud is more cost-efficient is related to the point of this paper which is on reproducibility. Also please have a look at Amazon webpage:
https://docs.aws.amazon.com/dlami/latest/devguide/gpu.html
Here, they explicitly mention that "A GPU instance is recommended for most deep learning purposes. Training new models will be faster on a GPU instance than a CPU instance. You can scale sub-linearly when you have multi-GPU instances or if you use distributed training across many instances with GPUs.",  so having this experiment again neither back up the arguments in the paper, nor add value to the paper.
* The major issue with this submission is that the experiments are not related to the arguments of the paper, and are not conveying any message towards backing up the arguments of the paper.
* Another crucial problem is that to allow a fair comparison especially in neural models, as shown in several studies(see [1] as a sample), this is important to account for random seeds and study how it impacts the model performance, to allow a fair evaluation of the models this is important to consider this factor, fair evaluation of models is argued to be the main point of this paper, however, the authors does not consider this factor in the paper, nor study it in the experiments.
[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers and Iryna Gurevych


Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3-or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code.

The motivation of the paper is to be able to train low precision networks to a high-accuracy. Quantization is a useful tool in model compression, and doing it well for very low-precision models (2-3 bit precision specifically), is challenging.
The main contribution of the paper comes from:
a) Step Size Gradient: They propose a gradient which is sensitive to the distance between the value and the transition point. This is different from other methods which have gradients dependent only on the clip point.
b) Step Size Gradient Scale: This is an interesting contribution, where they try to match the ratio of average update of the step size ‘s’ and average magnitude of ‘s’, with that of the network weights. This leads them to scale the gradient according to the precision and number of parameters. They demonstrate that this scaling actually helps improve the accuracy.
The results for 8-bit precision are not new. Several results (Quantization and Training of Neural Networks for Efficient
Integer-Arithmetic-Only Inference, Jacob et al., Quantizing deep convolutional networks for
efficient inference: A whitepaper, Krishnamoorthi et al.), show 8-bit quantization results where the accuracy matches floating point accuracy, and in some case exceeds it (low precision quantization acting as a regularizer). However, the results for lower precision are impressive.
There are a few questions:
1. In sec 2.1, you mention that ‘each layer of weights and activations has a distinct step size, represented as an fp32 value, initialized to …’. Can you explain the intuition behind the initial value of the step size, and how is it a function of v?
2. ‘Model Compression via Distillation and Quantization’ (Polino et al.) shows distillation actually helps significantly improve accuracy. I wonder if the authors have tried different weight combinations for the distillation loss, and using bigger models as teacher models.
3. I would like to get more details of the inference setup, specifically the size and inference latency improvements over full-precision networks. The practical applicability of low-precision networks, specifically 2-bit and 3-bit networks, equally depends on the inference infrastructure, as it does on the training improvements.
4. Have you evaluated your method for a non-Vision usecase?
Overall this is a good work, I would tend towards accepting this.


We analyze the Gambler's problem, a simple reinforcement learning problem where the gambler has the chance to double or lose their bets until the target is reached. This is an early example introduced in the reinforcement learning textbook by Sutton & Barto (2018), where they mention an interesting pattern of the optimal value function with high-frequency components and repeating nonsmooth points. It is however without further investigation. We provide the exact formula for the optimal value function for both the discrete and the continuous case. Though simple as it might seem, the value function is pathological: fractal, self-similar, zero-derivative almost everywhere, not smooth on any interval, and not written as elementary functions. It is in fact one of the generalized Cantor functions, where it holds a complexity that has been uncharted thus far. Our analyses could lead insights on improving value function approximation, gradientbased algorithms, and Q-learning, in real applications and implementations.

The paper derives the optimal value function for the continuous and discrete versions of the Gambler's problem. It also explores certain properties of this value function and finds that it has many fractal-like qualities.
This paper focuses on an unconventional but intriguing topic, bringing more insight into the possible shapes of value functions. The main result, which describes the value function for the gambler's problem, seems surprisingly complex and could have some practical implications for value-learning. Unfortunately, I find the paper is difficult to follow due its organization and the presence of many typos and unclear phrases. In summary, while I believe the topic is interesting and the work seems sound, the presentation would need to be improved significantly for me to recommend acceptance.
Here are certain points to be addressed:
1) On p.2, "The optimal value function presents its self-similar, fractal and non-rectifiable form". It is not clear from Thm. 11 that these properties hold. Some further explanation would be helpful
2) I think it would be helpful to include some descriptions of certain concepts that may be less familiar to the reader in the appendix. E.g. the Cantor function
3) Fig.1 should have larger labels and dots. It is difficult to see currently.
4) p.3 "the similar fractal patterns observed empirically in other reinforcement learning tasks." Certain concrete examples would be helpful here since I was not aware that this was common.
5) p.4 The paragraph on 'self-similarity' was not clear to me. What is meant by 'chaos' or 'dimension' in this context?
6) p.5 "Otherwise an arbitrary small amount of will have a fixed probability of reaching the target 1." I am unsure what this sentence means.
7) Perhaps starting with the intuitive description of v(s) (the recursive form) first before presenting the closed form solution would be easier to follow.
8) Why is there no discussion/conclusion section at the end?
There are numerous typos scattered throughout the paper. Here are some from the first page:
- abstract: "where they mention an interesting pattern of
the optimal value function with high-frequency components and repeating nonsmooth
points but without further investigation." Awkward phrase
- abstract: "With the analysis," Unnecessary phrase
- par. 1: "is described as below" -> "is described below"
- "which would hide its attractiveness." Awkward phrase
- "as an representative" -> "as a representative"
- "family of Markov decision process" -> "family of Markov decision processes"
- "the amount of bet." -> "the amount bet."
- "a round of bet." -> "a round of betting."
- "action-state value function" -> "state-action value function"
- "n as the starting capital (n denotes the state in the discrete setting)," The phrase is a bit confusing as it suggests n denotes both the discrete state and the starting capital.


The problem of learning from label proportions (LLP) involves training classifiers with weak labels on bags of instances, rather than strong labels on individual instances. The weak labels only contain the label proportion of each bag. The LLP problem is important for many practical applications that only allow label proportions to be collected because of data privacy or annotation cost, and has recently received lots of research attention. Most existing works focus on extending supervised learning models to solve the LLP problem, but the weak learning nature makes it hard to further improve LLP performance with a supervised angle. In this paper, we take a different angle from semi-supervised learning. In particular, we propose a novel model inspired by consistency regularization, a popular concept in semi-supervised learning that encourages the model to produce a decision boundary that better describes the data manifold. With the introduction of consistency regularization, we further extend our study to non-uniform baggeneration and validation-based parameter-selection procedures that better match practical needs. Experiments not only justify that LLP with consistency regularization achieves superior performance, but also demonstrate the practical usability of the proposed procedures.

Summary of the paper:  Learning from label proportions (LLP) is an area in machine learning that tries to learn a classifier that predicts labels of instances, with only bag-level aggregated labels given at the training stage.  Instead of proposing a loss specialized for this problem, this paper proposes a regularization term for the LLP problem. The core contribution of this paper is to use the idea of consistency regularization, which has become very popular in semi-supervised learning in the recent years.  The regularization term takes a perturbation of an input sample, and then force the output of the original and perturbed sample to be similar by minimizing  a KL divergence of the two output distributions.  Experiments show the performance of the proposed method under two bag generation settings.  The paper also finds empirically that the hard L_1 has high correlation with the test error rate, which makes it an ideal candidate when the user splits the validation data from training data (meaning there are no ground truth labels for each instances).
Reasons for the decision of the paper:  The proposed consistency regularization term seems to be borrowed directly from semi-supervised learning (SSL) research, and is not really specialized for the LLP problem.  Although the discovery that adding this regularization term for the LLP problem increases generalization performance is novel, using this term for data without ground truth labels in image datasets such as CIFAR10/100 and SVHN itself has low novelty, since these datasets satisfy the smoothness assumption used by consistency regularization methods.  Another issue is that there are only one trial for every experiment, making it hard for the reader to see if the results are statistically significant or not.  For these two reasons, it was hard to give a high score decision for this paper.  However, the paper’s motivation is interesting, and the direction to explore regularization techniques for the label proportion learning problem is important and seems novel for this area.  If the regularization method can be extended in a way that relates to the LLP problem, it would make the paper much stronger.
Other minor comments:
In the experiments, the average test accuracy of the last 10 epochs are reported.  I was curious if the last epochs have better models than earlier epochs.  In many weakly-supervised areas such as noisy labels, it is common that the accuracy goes up very quickly but then gradually decreases.  Does this also happen for the LLP problem?
In J_cons, does the expectation need to be over p?
***After author response:
Thank you for answering my questions!  Although I have a better understanding of the paper now, I still have the same concerns and would like to keep my score.