We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost. We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean of their states, namely, the mean-field state. For such a game, based on the Nash certainty equivalence principle, we provide sufficient conditions for the existence and uniqueness of its Nash equilibrium. Moreover, to find the Nash equilibrium, we propose a mean-field actor-critic algorithm with linear function approximation, which does not require knowing the model of dynamics. Specifically, at each iteration of our algorithm, we use the single-agent actor-critic algorithm to approximately obtain the optimal policy of the each agent given the current meanfield state, and then update the mean-field state. In particular, we prove that our algorithm converges to the Nash equilibrium at a linear rate. To the best of our knowledge, this is the first success of applying model-free reinforcement learning with function approximation to discrete-time mean-field Markov games with provable non-asymptotic global convergence guarantees.

Summary and Decision
This paper studied an actor-critic learning algorithm for solving a mean field game. More specifically, the authors showed a particular actor-critic algorithm converges for linear-quadratic games with a quantitative bound in Theorem 4.1. Notably, results on learning algorithms for solving mean field games without prior knowledge of the parameters are rare, so results of this type is highly desirable. However, the algorithm studied in this paper is uniquely tailored for the (very special!) linear-quadratic setting, which is very unsatisfying. We will discuss concern this in detail below.
Overall, I recommend a weak accept for this paper.
Background
Mean field games is a theory of large population games, where we assume each agent has infinitesimal contributions to the dynamics in the limit as number of agents go to infinity. Similar to mean field theory of particles, the limiting dynamics can be completely characterized by a single distribution of agents, commonly know as McKean-Vlasov dynamics. The theory drastically simplifies computation for large population games: while it is essentially impossible to find a Nash equilibrium for a 100 agent game, we can compute the Nash equilibrium for the mean field limit and approximate the finite game.
Mathematically, mean field games remain very difficult to solve even knowing the parameters and dynamics. Therefore it is often important to first study a simple case where we can solve the game analytically. In the context of optimal control and mean field games, we can often recover closed form solutions (up to the solution of a Riccati equation) when the dynamics are linear and the cost is quadratic. We call this class of games linear-quadratic mean field games (LQ-MFG). To interpret the LQ assumption, typical control problems in this setting can be recast into a convex optimization problem in the control (or strategy) using convex analysis techniques. Therefore LQ assumptions provides both theoretical and computational tractability.
Here we will specifically note the paper of Elliot, Li, and Ni, where we can find a closed form solution of the discrete time LQ-MFG with finite horizon.
https://arxiv.org/abs/1302.6416
Furthermore, we will also distinguish between games with a finite horizon and infinite horizon. While there are difficulties associated with both cases, typically an ergodic infinite horizon problem removes the time variable from the equation, making the problem slightly easier. Hence many researchers in MFG prefer to begin by studying the ergodic problem.
In the context of reinforcement learning, we are more interested in solving MFG without knowledge of underlying parameters, dynamics, or even the cost function. This direction is still relatively new for the MFG community, and many problems remain open. The ultimate goal of this line of research is to develop generic and scalable algorithms that can solve general MFGs without knowledge of the game parameters/dynamics/cost etc.
Discussion of Contributions
This work is the first analysis of actor-critic algorithms for solving MFG. At the same time, the paper studies discrete time MFGs, which is generally less popular but no less interesting. Therefore a theoretical convergence result in this setting is highly desired.
Overall, the mathematical set up of this problem is very convoluted. This likely motivated the authors to make more simplifying LQ type assumptions to recover stronger results. Even with these assumptions, to put all the pieces of the puzzle together is no easy task. The authors have to consider the interaction between the agent state and the mean field state, as well as the estimation of optimal controls and how to bound errors from estimation error. This led to a long appendix of proofs - while too lengthy to verify, the results seem sensible.
From this, I believe the mathematical analysis itself is a worthy contribution. This paper will serve as a good starting point for future analysis of more complex problem settings and other learning algorithms in MFGs.
Discussion of Limitations
The main concern regarding this paper is on quantifying how much of a contribution the results add to the broader community. While results on LQ-MFGs are always nice to have, I believe the specific actor-critic algorithm depends too much on the LQ structure for this work to be useful. Two examples of these are:
1. on page 5, above equation (2.1), the actor-critic algorithm will be only seeking policies that are linear in the state x. This is taking advantage of the fact that we know LQ-MFGs have linear optimal policies.
2. on page 7, below equation (3.7), the algorithm requires to know the form of the gradient of the cost function - and therefore leading to a direct estimation of matrices \Upsilon that form the gradient. This is only possible in LQ-MFGs.
Therefore, results from this paper will be very difficult to generalize to other actor-critic algorithms for LQ-MFGs. At the same time, it's also difficult to generalize these results to the same actor-critic algorithm for non-LQ-MFGs.
We also note that if we can assume knowledge of the LQ form of underlying dynamics and the form of the cost function, but no knowledge of the parameters, the problem reduces down to a parameter estimation problem. In this case, we can speculate the results of Elliot, Li, and Ni can be adapted to the ergodic case, and we can recover the approximate optimal controls given estimated parameters. Furthermore, in some sense, this particular actor-critic algorithm is implicitly estimating a sufficient set of parameters (the \Upsilon matrices) to find the optimal control. Essentially, if we rely too much on the LQ structure, the problem is then rendered much less interesting.
In summary, the ultimate goal of this line of research is to approximately solve non-LQ games, therefore the value of this current paper is very incremental in the larger context of learning MFGs. While serving as a reference for future analysis of related algorithms for MFGs, it will be difficult to borrow concrete ideas and results from this paper to build on.
*Edit:* I believe the discussions below have address several important concerns, and I will raise my score to accept.


In the field of Continual Learning, the objective is to learn several tasks one after the other without access to the data from previous tasks. Several solutions have been proposed to tackle this problem but they usually assume that the user knows which of the tasks to perform at test time on a particular sample, or rely on small samples from previous data and most of them suffer of a substantial drop in accuracy when updated with batches of only one class at a time. In this article, we propose a new method, OvA-INN, which is able to learn one class at a time and without storing any of the previous data. To achieve this, for each class, we train a specific Invertible Neural Network to extract the relevant features to compute the likelihood on this class. At test time, we can predict the class of a sample by identifying the network which predicted the highest likelihood. With this method, we show that we can take advantage of pretrained models by stacking an Invertible Network on top of a features extractor. This way, we are able to outperform stateof-the-art approaches that rely on features learning for the Continual Learning of MNIST and CIFAR-100 datasets. In our experiments, we reach 72% accuracy on CIFAR-100 after training our model one class at a time.

[update after rebuttal]
I thank the authors for their detailed reply, answers to my questions, and updates to the paper. (I especially appreciate the substantial effort to address all points and improve the manuscript even given the strongly negative rating. I am not personally in favour of "extremifying" the rating system the way ICLR did this year; I think it discouraged many authors from working on their rebuttals, and caused many reviewers to have a sense of inertia to keep their bad scores. I'm glad the authors didn't fall into the first camp, and I'm doing my best not to fall into the second! I think you did a very good job of improving the paper).
I think the manuscript has improved substantially; especially the clarifications about (not) using negative samples and additional detail on INNs with citations.
I find some of the sentences still unclear, and strongly suggest having the text read over for grammar/clarity for a camera-ready version.
But overall I recommend acceptance of this work.
------------
Paper summary:
This paper does continual learning using an invertible neural network (INN) trained to recognize each class. At test time a new example is presented to all of the INNs and the INN whose prediction has the smallest norm is used to predict the class of the test example. They do experiments on MNIST and CIFAR-100
Paper contributions:
- Review of methods and evaluation settings for continual learning
- Review of invertible neural networks
- Experiments comparing the proposed method to several other continual learning methods
- Examination of memory cost
- Exploration of feature space of trained INNs
Review summary & decision:
The proposed idea of using invertible networks for continual learning sounds interesting, and I think that this could be a good paper. The experimental evaluation is not consistent or complete enough for me to to tell, however, and the core motivation for the idea is not clearly explained. There are also some less critical, but still important aspects of the paper (related work, clarity of explanations, repetitiveness) which lead me to decide this paper is not currently ready for publication.
Reasons for decision:
1. A lot of statements and decisions are made without being explained, or are unclear / innacurate
      - In CL, the objective is to learn several tasks one after the other" this is not the objective, it's the problem setting. The objective is classification accuracy (or some other metric) in this setting.
      - The choice of datasets seems a little odd; why not CIFAR-10 as well?
      - It's very misleading to say your method is able to learn one class at a time without storing data; unless I've misunderstood this is only after pretraining all of the invertible networks with labelled data.
       - Why 500 examples per class for CIFAR-100, and how many for MNIST? What criteria was used to arrive at this number?
      - in the abstract you state the definition of continual learning says you can't have access to the data from previous tasks, but then mention that previous works use samples from previous data. This seems contradictory
      - I'm not extremely familiar with Li & Hoeim (learning without forgetting), but I don't think it's very accurate to call it distillation regularization
      - In 4.1 I don't think it's accurate to call this compression; it's just having smaller size layers in between (unless I misunderstood).
      - I'm not familiar with model superposition; it should be more explained since your claims of having lower memory cost rely on this, as far as I can tell.
      - The experimental baselines seem inconsistent / not comparable, making it difficult to evaluate what is going on. E.g. some update pretrained features and some do not. Some (including yours) do offline pretraining on a subset of examples, with inconsistent amounts of data, and sometimes this is considered part of training (Fearnet) and sometimes not (yours).
       - I don't understand the central motivation for using INNs. The only sentence that seems to talk about it is "this way ... network won't be able to have an output similar to the outputs on data from its training set", but this is not clear, not enough explanation, and seems somewhat innaccurate. It's not that it's not able, it's just not likely (I think; please correct me if I'm wrong!), and I don't see why this isn't also true of e.g. MLPs (i.e. for MNIST, train 10 MLPs each to classify [class i] vs [all classes except i] - In order to justify the proposed approach, the baseline described above (using MLPs instead of invertible NNs) would be very useful
 2. The training regime for the INNs is not clear (are they shown negative (other class) examples? How many, how are they sampled? I didn't find any hyperparameters for the INN training in the appendix, even though it says they should be there.
 3. The single head setting seems very similar to (or maybe the same as?) open set learning; I think this should be mentioned and works in this area should be reviewed.
 4. Section 3 is repetitive and unclear. It could be greatly shortened to make space for more experiments.
 5. It would be nice to see computational cost as well as memory cost; this is important in many settings where continual learning would be deployed.
Feedback/suggestions/nits (not necessarily part of decision assessment):
1. Cite the definition of continual learning (e.g. with a reference to a textbook or review)
2. A lot of the writing is unclear, wordy, and/or grammatically incorrect
    - inconsistent verb tense
         -  e.g. "if one would need .... it is required" should be "if one would need .... it would require" I'd suggest rewording this sentence entirely, because it's misleading - it says "retrain on this new dataset (which sounds like train just on the new data), but I guess you mean retrain on all data including the new data. e.g. "Updating the model with new data requires retraining on the full dataset (old + new data). However, there are.... method may not be applicable."
         - "we are reaching" -> we reach    - frequent use of "indeed" when it doesn't make sense
    - Section 3 repeats the intro, 3.1 and 3.2 are sort of saying the same thing, and are also sort of repetitive of the related work. The related work should all be in the related work section, and this section should just be about what _you_ are doing. Invertible neural networks should be reviewed in something like a 'background' section, which possibly could be in appendix. Right now that section has even more related work in it
3. Include tsne hyperparameters in appendix
4. "appendix" not "annexe" (annexes are on buildings :) )
5. when you define continual learning, bold it (not italicize)
6. "Y is then Y_i" -> Y = Y_i"
7. Caption for Figure 3 is unclear; I don't understand what I'm supposed to take away from these images. I also don't see a black cross.
Questions:
1. What is the motivation for using INNs specifically? (rather than e.g. normal MLPs).
2. What is the training regime for the INNs?
3. What is model superposition, is it expensive in computation or something other than memory, and if you don't do that what is the memory cost of your method?


We apply a reinforcement learning (RL) based approach to learning optimal synchronization policies used for Parameter Server-based distributed training of machine learning models with Stochastic Gradient Descent (SGD). Utilizing a formal synchronization policy description in the PS-setting, we are able to derive a suitable and compact description of states and actions, allowing us to efficiently use the standard off-the-shelf deep Q-learning algorithm. As a result, we are able to learn synchronization policies which generalize to different cluster environments, different training datasets and small model variations and (most importantly) lead to considerable decreases in training time when compared to standard policies such as bulk synchronous parallel (BSP), asynchronous parallel (ASP), or stale synchronous parallel (SSP). To support our claims we present extensive numerical results obtained from experiments performed in both simulated and real cluster environments. In our experiments training time is reduced by 44% on average in simulated and 28% in real cluster environments, respectively. Moreover, the learned policies generalize to multiple unseen circumstances.

This paper studies how to improve the synchronization policy for parameter server based distributed SGD algorithm. Existing work focus on either synchronous or asynchronous policy, which often results in straggler or staleness problems. Recent research proposes different ways to make the policy design fully adaptive and autonomous. This paper proposes a reinforcement learning (RL) approach and shows promising results to reduce the total training time on various scenario.
A major challenge is to design the state and action spaces in the reinforcement learning setting. It requires the design can be generalized to different scenario, while ensuing efficient policy learning. Compared to existing policies such as BSP, ASP and SSP, RL has an advantage to adapt to non-stationary situations over the training process. Compared to other existing approaches, RL could be fully data-driven.
The paper formalizes an RL problem by minimizing the total training time to reach a given validation accuracy. To minimize the number of actions, only 3 actions coming from BSP, ASP and SSP are used. The state space is designed to capture similar loss curves, and to be independent of the number of workers (as much as possible). A policy network is used make decisions after trained with exiting methods.
Numerical results validate that the RL policy improves the training time compared to BSP, ASP and SSP. Different number of works and models, dataset are also tested to show the RL policy is generalizable to unseen scenario. Although all the results are simulated in a controlled environment, Figure 4 gives a very interesting illustration showing the advantage of using the RL policy. I still have detailed comments (see below), but I find the paper well written, and the author(s) has obtained promising results.
Detailed comments:
-	The validation accuracy 88% on MINST seems pretty low to me to stop the algorithm, in particular when training multiple layer neural networks. What would happen if the accuracy is increased, can the RL approach still find a good policy? What about the validation accuracy on CIFAR-10?
-	I still have some concern of the computation time obtain the RL state per step. In particular, the time cost to compute the loss L on different weights w. How do you address this issue? Is L computed on the validation set? What is its size? This parameter seems to me highly sensitive when the policy is used for different dataset, in particular the dataset vary. It would be better to have more discussions in the paper or appendix.
-	What is the final test accuracy on the trained models using different policies? This allows us to see whether the approach has not over-fitted to the training/validation set.
Some typo:
Page 2 line 2 AP -> ASP
Page 5, last line 4: converge -> convergence


Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-102 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-102 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-102 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.

--- Updated during response period ---
Authors successfully answers all my questions. I revise my rating to Accept.
-----
Summary:
This paper proposes another benchmark dataset for neural architecture search. The idea is following the NASBench-101 dataset, that in a given search space, densely sampled all existing architectures and train each of them on three tasks for multiple times, and using the obtained metrics as a tool to evaluate an arbitrary neural architecture search algorithm. The paper also presents comprehensive reports on the statistics, revealing a strong performance correlation between tasks, and evaluate some baseline NAS algorithms.
I think this paper will be valuable to the research community for these reasons: (1) the dataset contains a more geologically complex search space comparing to the original NASBench-101, whose search space is restrained in certain ways; (2) released metrics include more meaningful information rather than single point value in NASbench; (3) it uses 3 datasets rather than 1.
My major concerns, which I will detail later, is the phrasing "algorithm-agnostic" does not truly reflect the difference between their approach and NASBench-101, and about the architecture search space design details.
Altogether, I think even the technical novelty is incremental, the work is not trivial considering the computational cost. I am willing to improve my score if my concerns are addressed during the rebuttal period. Nevertheless, this dataset is a strong subsidy of existing NASBench-101 and can benefit the research community and serves as an important baseline to evaluate a NAS algorithm.
Strength
+ Clear motivation to use an operation-on-the-edge search space that is widely used in NAS domain.
+ Extensive experiments on evaluating 15K architectures over 3 datasets
+ Detailed statistics on the search space
+ Good baseline experiments comparison
Main concerns about this dataset:
- Comparing to NASBench-101 in terms of "Algorithm Agnostic", it is in a "more-or-less" game but not a "yes-or-no" one, so that AA-NAS-Bench does not seem appropriate. In my perspective, this dataset has not shown significant differences for the following reasons.
1. With proper adaptation, both NASBench-101 and the one in this paper are "algorithm agnostic". For example, original ENAS is training a reinforcement learning sampler that learns to predict a string with encoding [id1, op1, id2, op2] for each node, where id1, id2 is the IDs of the previous node to connect, op1, op2 is the operation choice for each edge. Since NASBench has operation on output node, one could simply make RL sampler to predict [id1, id2, op1], or another string encoding that suits the search space better. In my perspective, Ying et al. mentioned that many NAS algorithms cannot be directly evaluated on NASBench-101 are because the search space is different, but it does not mean using NASBench-101 is impossible. On the other hand, for some other state-of-the-art algorithms, like Proxyless-NAS on ImageNet, the search space is also different from the one proposed in this paper, but likewise, it does not indicate evaluating Proxyless-NAS on this dataset is impossible.
2. NASBench-101 does impose constraints on maximum edge number equals to 9 with 7 nodes in their space and results in 423K architectures. However, this constraint is no longer applied if you reduce the number of node to 6 (i.e. all possible architectures can be sampled), yet it still contains around 64K architectures, which is more than 15K in the proposed dataset. In this perspective, NASBench is a larger dataset and "algorithm agnostic".
To summarize, I acknowledge the paper's contribution is using an operation-on-the-edge search space that is widely used in previous NAS algorithms while NASBench-101 is using operation-on-the-node space. However, it only makes the proposed dataset "more algorithm agnostic" with less effort, and it does not make the previous NASBench-101 "not" algorithm agnostic. If using the current name AA-NAS-Bench, I think it is not fair for the NASBench-101, specifically they are 4 times larger after removing the edge number constraints.
- Questions about architecture space design
1. Why using average pooling instead of max pooling?
2. How do you compute the total architecture number 15,625 in Table 3? In your setting, with the number of node V=4 densely connected DAG, it should have 6 edges as depicted in Figure 1, and each edge has 5 possible operations, i.e. total number = 6^5 = 7776. I am confused about this point, could author comment more on this number?
3. Is there any topologically equal architectures in this space? For example, let's name the node 1,2,3,4, and the following two architectures should be the same since input edges are summed before passed to the next node. I listed **non-zeroed** edge as, id1->id2: op
Architecture 1:
1->2: conv3x3
2->4: skip
1->3: conv1x1
3->4: skip
Architecture 2:
1->2: conv1x1
2->4: skip
1->3: conv3x3
3->4: skip
If the pruning is not effectively conducted, my worry is the actual number of architectures is smaller.
Minor comments:
1. DARTS results on the are quite poor as mentioned in this paper that, DARTS will eventually converge to an architecture with all skip connection. However, it could be a simple fix, by tracking the architecture evolution during the search and report the best like early-stopping. Will this improve DARTS results?
2. Since ENAS is the first work using parameter sharing on the NAS problem, could the author add it to the baseline?
3. In table 4, what is the average (94.37 for CIFAR-10) mean in the "optimal" column? Is this the mean performance of all architectures? If so, it is quite strange to see all the baselines are selecting architectures worse than the average performance. Or it is the best architecture performance as indicated in the caption? This "average" column for "optimal" seems confusing.
4. The dynamic ranking of architecture in Figure 5 is very interesting. Architecture ranking seems stable after the 190th epoch. Could the author provide another visualization, showing when stabilization happens in between epoch number 150 and 190?
5. Figure 4, correlation matrix for top 4743 architectures are significantly lower than the full and 1387 ones, is this possible because of repetitive architectures in the space are not pruned? And, what is the reason for number 4743 and 1387?
6. ResNet (star in Figure 2) seems to perform very well. Does this indicates the proposed search space is not much meaningful, considering there are only 1~2% for NAS to improve?


Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period. The basic motivation behind these methods is that orthogonal transformations are isometries of the Euclidean space, hence they preserve (Euclidean) norms and effectively deal with vanishing/exploding gradients. However, this ignores the crucial effects of non-linearity and noise. In the presence of a non-linearity, orthogonal transformations no longer preserve norms, suggesting that alternative transformations might be better suited to non-linear networks. Moreover, in the presence of noise, norm preservation itself ceases to be the ideal objective. A more sensible objective is maximizing the signalto-noise ratio (SNR) of the propagated signal instead. Previous work has shown that in the linear case, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure. Motivated by this finding, here we investigate the potential of non-normal RNNs, i.e. RNNs with a non-normal recurrent connectivity matrix, in sequential processing tasks. Our experimental results show that non-normal RNNs outperform their orthogonal counterparts in a diverse range of benchmarks. We also find evidence for increased non-normality and hidden chain-like feedforward motifs in trained RNNs initialized with orthogonal recurrent connectivity matrices.

The focus of this paper is on exploring non-normal initializations for training vanilla RNN for sequential tasks. They show on 3 different tasks, and a real-world LM task that  non-normal initializations of vanilla RNNs outperform their orthogonal counter-parts when particular forms of initialization are considered.
Although the results for sequence task do not outperform the gated counterparts, the authors present an interesting exploration of initializing non-normal RNNs that outperform the orthogonal counterparts. It is good to see this line of work being explored as an alternative to exploring more complex architectures with many more parameters than necessary for the task.
Strengths:
    1. The paper explores non-normal RNNs and demonstrates on  3 synthetic tasks - copy, addition and pMNIST - how with careful initialization the proposed approach outperforms their orthogonal initialization counterpart. This line of experimentation is interesting as it potentially opens the door for more expressive modeling for sequential tasks by expanding the solution space of the weight matrices being learnt i.e orthogonal matrices are a special case.
    2. The authors do a great job in motivating the paper, and the explanation is clear and easily understandable. The toy simulations in Section2.2 really helps drive the reasoning behind why chain initialization improves over orthogonal initialization.
    3. Based on the insight from trained RNNs where the trained  weights exhibit a chain like structure, the authors attempt to modify the LSTM gate initializations well. However, they do not see any specific gain by doing so, and moreover they show analysis that demonstrate that the LSTM gates do not learn these chain like structures. However, they do have insight into the regularities of these learnt weights which could potentially open the door for more interesting initialization methods for training such gated architectures.
Issues to be addressed in the paper:
1. The plots are quite small and hard to follow. Can the authors enlarge these so they span the entire page? Also, for pMNIST it would be good to provide accuracy scores as well as a function of the training epochs.Finally, it would be good to include a comparison against LSTMs (and even Transformer networks) so it is easier for the reader to see where these approaches stack against architecture changes.
2. The authors are missing a reference to this  work - http://proceedings.mlr.press/v48/henaff16.pdf  - which provides empirical analysis for the 3 synthetic tasks to test the ability of vanilla RNNs for solving long span sequential tasks.
3. What about stability of these non-normal RNNs? For example, if we perturb the inputs to the training for the LM task how much variance do we see in the performance of these models?


Modelling highly multi-modal data is a challenging problem in machine learning. Most algorithms are based on maximizing the likelihood, which corresponds to the M(oment)-projection of the data distribution to the model distribution. The M-Projection forces the model to average over modes that can not be represented by the model. In contrast, the I(information)-Projection ignores such modes in the data and concentrates on the modes the model can represent. Such behavior is appealing whenever we deal with highly multi-modal data where it is more important to model single modes correctly instead of covering all the modes. Despite this advantage, the I-projection is rarely used in practise due to the lack of algorithms that can efficiently optimize it based on data. In this work, we present a new algorithm called Expected Information Maximization (EIM) for computing the I-projection solely based on samples for general latent variable models, where we focus on Gaussian mixtures models and Gaussian mixture of experts. Our approach applies a variational upper bound to the I-projection objective which decomposes the original objective into single objectives for each mixture component as well as for the coefficients, allowing an efficient optimization. Similar to GANs, our approach also employs discriminators but uses a more stable optimization procedures optimizing a tight upper bound. We show that our algorithm is much more effective in computing the I-projection than recent GAN approaches and we illustrate the effectiveness of our approach for modelling multi-modal behavior on two pedestrian and traffic prediction datasets.

The paper presents an algorithm to match two distributions with latent variables, named expected information maximization (EIM). Specifically, EIM is based on the I-Projection, which basically is equivalent to minimizing the reverse KL divergence (i.e. min KL[p_model || p_data]); to handle latent variables, an upper-bound is derived, which is the corresponding reverse KL divergence in the joint space. To minimize that joint reverse KL, a specific procedure is developed, leading to the presented EIM. EIM variants for different applications are discussed. Fancy robot-related experiments are used to evaluate the presented algorithm.
Overall, the paper is in good shape wrt the logic and the writing. My main concerns focus on the novelty (compared to existing methods that are similar but not discussed) and the experiments. For the former, reverse KL has been exploited before, both in the marginal space [1] and the joint one [2]. Other detailed comments are listed below.
As Eq 4 is for matching two joint distributions, discussions/comparisons should be made to reveal the novelty of the presented EIM over existing methods such as [2], etc.
In Figure 2 (b), the experimental settings for adversarial learning are not fair, as the discriminator is not fixed there.

In Sec 4.4, it seems EIM is highly overlapped with VIPS. So what're the advantages of EIM here?
In Figure 3, how many steps for Generator and Discriminator are used for f-GAN? Does f-GAN finally converge? It would be helpful if some results are given to demonstrate the final state of each method.
In Eq. 9, adding the denominator q(z_i) will change the optimal solution. Why only add it to the first term?
In Section 5.3 and Figure 5, “SSD” might be a typo.
[1] Adversarial Learning of a Sampler Based on an Unnormalized Distribution. AISTATS 2018.
[2] Symmetric Variational Autoencoder and Connections to Adversarial Learning. AISTATS 2019.


In this paper, we propose an end-to-end graph learning framework, namely Iterative Deep Graph Learning (IDGL), for jointly learning graph structure and graph embedding simultaneously. We first cast graph structure learning problem as similarity metric learning problem and leverage an adapted graph regularization for controlling smoothness, connectivity and sparsity of the generated graph. We further propose a novel iterative method for searching for hidden graph structure that augments the initial graph structure. Our iterative method dynamically stops when the learned graph structure approaches close enough to the optimal graph. Our extensive experiments demonstrate that the proposed IDGL model can consistently outperform or match state-of-the-art baselines in terms of both downstream task performance and computational time. The proposed approach can cope with both transductive training and inductive training.

The paper proposes an extension of learning graph structure and GNN concurrently, by considering that real-world graphs are often noisy and incomplete. The idea of optimizing the intrinsic graph structure iteratively for down-stream prediction tasks is interesting. Experimental results demonstrate the effectiveness of proposed method.
Strengths:
1）the paper proposes a learnable similarity metric function and a graph regularization for learning an optimal graph structure for prediction.
2）Besides raw node features, the paper attempts to optimize graph structures via learned node embeddings in an iterative manner.
3）The paper is easy to read, and experiments show that the proposed method performs well.
Weaknesses:
1）Compared with LDS [1], this work seems to overlook the bi-level optimization problem for learning model parameters based on the optimal graph structure. The reason behind this method is expected.
2）Although the paper claims that the dependence of raw node features for learning graph structure has been weakened,  empirical analysis on this point is not given. The feature matrices in experiments are not strictly independent with graph structures.
3) As shown in Appendix B, too many hyper-parameters are involved. I conjecture it will be difficult to reproduce the experimental results.
4) Eqs.(2), (3) and (10) are problematic. Node embeddings Z should be included in them. Eq.(10) does not have theoretical proof. According to Eq.(10), the method cannot handle graphs with noisy edges. In experiments, there are edge deletions, but no edge addings. Experiments with attacked graph are expected.
5) Although this method is claimed efficient, it is indeed slower than the classic GNNs due to the iterative operation. The details of training time comparison between this method and GNNs such as GCN and GAT will be helpful. I was wondering why this method is faster than LDS. Is it due to removing the bi-level optimization problem ?
6) Although the method can handle inductive training, it is hardly scale to big networks. Pubmed is an open citation network with around 20,000 nodes similar to Cora and Citeseer. Those three datasets are popularly used in GNNs as testbed. However, Pubmed is not used in this work. I conjecture that the new method cannot handle such a big dataset efficiently.
Overall, this proposed method is well motivated, but the technical novelty is limited.


We define a goodness of fit measure for generative networks which captures how well the network can generate the training data, which is necessary to learn the true data distribution. We demonstrate how our measure can be leveraged to understand mode collapse in generative adversarial networks and provide practitioners with a novel way to perform model comparison and early stopping without having to access another trained model as with Frechet Inception Distance or Inception Score. This measure shows that several successful, popular generative models, such as DCGAN and WGAN, fall very short of learning the data distribution. We identify this issue in generative models and empirically show that overparameterization via subsampling data and using a mixture of models improves performance in terms of goodness of fit.

This work proposed a new goodness of fit measure for generative network evaluations, which is based on how well the network can generate the training data. The measure is zero if the network could perfectly recover the training data, and would represent how far it is from generating the training set in the average manner of the total least square sense, where the one-to-one mapping between the generated data and the training sample is constructed through latent space optimization. Using the proposed measure, the authors showed an interesting trend present in the DCGAN training and the impact of the residual connection. The authors might want to add some discussion in Section 4.2 regarding why the residual connection is detrimental for covering the support.  Increasing the model complexity through larger latent space dimension and learning mixtures is proposed as solutions to improve the measure as well.
With all the interesting results presented, I still have the concerns about the sensitivity of the proposed measure:
- It is an average over the training data or the selected sample. Above Section 4, the authors argued that "\hat{F}(G) > 0 meaning that we do not observe any memorization". This seems overly assertive. Since the measure is an average over the training data, it has difficulty to differentiate between one network which has almost zero value for part of the training data but large values for the rest, and another network with roughly the same \hat{F}(G) value but small values for all training data. The variance could help, but can not resolve this issue. This would be more important when the training data contains noise or outliers.
- It only concerns the generation of the training data, but not the sampled data from the network (at least not directly). Therefore it has no direct control of the fidelity of the generated samples.
- As shown by the authors, the proposed measure can be considered as the approximation of the true probability support not covered by the generative models, which also defines a necessary condition to avoid mode collapse. But what about the other part? It would have difficulty comparing two models with the same support but different high-density areas. Indeed, there are existing works which consider both the precision and recall of the generative models [1, 2, 3], and directly work with the generated samples instead of the training data. These should be discussed and compared with, not just the FID scores which have already been shown to have issues [3].
Some notations:
- In the last equation on Page 2,  should it be L_{G} instead of L_{D}?
- In the first equation on Page 3, should the denominator be N_{B} instead of N_{N}?
- "Optimality" in terms of generative models may depend on the downstream tasks. I do not think there exists a universal definition of "optimality" for generative models.
[1] M.S.M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. NeurIPS 2018.
[2] L. Simon, R. Webster, and J. Rabin. Revisiting precision and recall definition for generative model evaluation. ICML 2019.
[3] T. Kynkaanniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Arxiv:1904.06991.


Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task transfer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for a large-scale language representation model. Through experiments on SQuAD and the GLUE benchmark suite, we show that proximal pruned BERT keeps high accuracy for both the pretraining task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides a new perspective to help us analyze what large-scale language representation might learn. Additionally, RPP makes it possible to deploy a large state-of-the-art language representation model such as BERT on a series of distinct devices (e.g., online servers, mobile phones, and edge devices).

Models such as BERT are pretrained language models which provide significant improvement for different tasks, however they suffer from high huge size and complexity. This paper has proposed using proximal gradient descent to find sparse weights for BERT to reduce the number of parameters and make the model smaller. They concentrate on the drawbacks of the previous sparse-based approaches and claimed that they have convergence issues (they have provided some evidence in the appendix). therefore, they propose to use reweighed sparse method and optimise it using proximal gradient descent which provides a closed form solution for sparse constraint.
ALthough proposing a minor novelty (reweighted sparse optimization ), they have provided interesting results for both pretrained structure and fine-tuning for several different tasks. they have also provided some visualisation for the weight matrices after sparsification.
Their results are notably stronger than simply adding the L1 regularizer to the optimisation method.
The paper is well written and easy to follow with nearly comprehensive related work.
However, there are some drawbacks:
1. They have claimed that “ To the best of our knowledge, we are the first to apply reweighted l1 and proximal algorithm in the DNN weight pruning domain, and achieve effective weight pruning on BERT. ”, however proximal optimization has been used for DNN in works like “Combined Group and Exclusive Sparsity for Deep Neural Networks, 2017”.
2. It should be explained clearly about all the matrices included in the sparsification steps, despite only saying “parameters of the model”.
3. More analysis is required on the results, specially the diagrams for fine-tuning over different datasets.
4. It is essential to compare the method with other related works for Bert and transformer compression, including quantisation-based, factorisation-based, pruning, knowledge distillation papers such as:
--Prato, Gabriele, Ella Charlaix, and Mehdi Rezagholizadeh. "Fully Quantized Transformer for Improved Translation." arXiv preprint arXiv:1910.10485 (2019).
--Tang, Raphael, et al. "Distilling Task-Specific Knowledge from BERT into Simple Neural Networks." arXiv preprint arXiv:1903.12136 (2019).
--Sanh, Victor, et al. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv preprint arXiv:1910.01108 (2019).
--Ziheng Wang, et al. "Structured Pruning of Large Language Models."


In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems. Following assumptions made by prior works, we first discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced. Specific design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed. In addition to the gain gates, we further introduce overshoot gates for compensating insufficient step size in LISTA. Extensive empirical results confirm our theoretical findings and verify the effectiveness of our method.

1. Summary
The authors propose extensions to LISTA with the goal of addressing underestimation (by introducing “gain gates”) and including momentum (by introducing “overshoot gates”). The authors provide theoretical analysis for each step of their LISTA augmentations, showing that it improves convergence rate. Their theoretical statements are empirically validated and then a numerical comparison is performed between the most interesting LISTA variants. Their proposed GLISTA performs favorably, especially for networks of depth greater than 10.
2. Decision and arguments
The theory given is quite comprehensive and seems solid. Moreover it’s motivated by and helps real problems, namely L1’s well-known underestimation and LISTA’s lack of momentum. On top of that, the numerical results not only show your GLISTA outperforming others, but that by adding gates to other LISTA variants you can get better results. On the other hand (see details in the Questions section), there is some critical information missing from the empirical section which makes the results non-repeatable.
Training is not described, i.e. hyperparameter searches and stopping criteria. A couple of the plots validating theory are difficult to understand. The datasets used are synthesized and tiny, only 1000 samples (training/testing/validation sets are not described) and 250x500 dimensions. That means your network has many more parameters than data samples unless I am mistaken.
So I have given weak reject because the paper has strong theory but weak experiments making it hard to trust the conclusions. I really want to hear back about the questions raised below.
3. Questions
a) Just before eqn 12 you say that gains greater than 1 can be more appropriate. Am I understanding this correctly: *before* shrinkage you want to apply a gain on code elements that are “truly” nonzero, in order to cancel out the imminent L1 penalization? And you will learn to predict those code elements via parameters Lambda?
b) Just before Section 4, you describe the difference between yours and Moreau & Bruna’s momentum taps. If I understand correctly they have a matrix called W_m^(k) which is multiplied onto the previous iterate—but for each layer / time unit (k), the matrix may vary by learning (note that in the main body of their paper they don’t explicitly use ^(k) notation, but it is explained to be the same as the other LISTA parameter matrices which vary with time, and is made explicit in the appendix). So I think their momentum is in fact time-varying. Moreover it is certainly dependent on the previous iterate, using the function f(z^(k-1)) = W_m^(k) * z^(k-1). Did you mean something else by that? In what sense do you have a higher capacity—do you have more parameters?
c) If I understand correctly, in Figure 5a, 5b, for each point you have trained independently an N-layer GLISTA. Then you observe properties of your learned parameters.
For 5a: how do you compute the “output” of the gain gates? What is the input that gives this output?
For 5b: which ‘t’ is used to calculate ||W(t)D-I+U(t)A||? Or did you just train a single 15-layer GLISTA, and the x-axis is just ‘t’?
d) Could you tell us more about the hyperparameter tests for every method? How do we know it was a fair comparison? There are no error bars, but from experience I know that training LISTA type networks can be a pain. What algorithms did you use? Stopping criteria? Some plots start at zero layers (4c, 5b, 6a-c) and some start at 1 layer…
e) Interesting that in your real-data example, the sparse vector e has more non-zeros than zeros, is that really sparse? What dictionary A did you use, then? By your own description this task does not fit the sparse coding model you have analyzed…. Am I misunderstanding?
4. Additional feedback/ minor comments
a) Add reference for DOA estimation application
b) You should put something like “we prove this in the supplement” for props and theorems. Otherwise to readers less familiar with the literature it will seem like you forgot to put a reference.
c) It seems unnecessary to put ||epsilon||<= 0 , instead of epsilon=0.
d) Basically I think you should explain what your gate is before providing proofs about it. It would be useful to see a plot of the function g and/or kappa so provide some intuition about what you are doing to the architecture (nonlinear? Linear? Threshold? Etc.). Capital “Lambda” is not defined until after theorems are provided about the functions g, kappa, f. In fact it seems like kappa is only useful for the sake of proofs. The first time through, I over-read the statement “All the learnable parameters are thus collected as Lambda…” just before Section 3.1.1 because I was thinking “Why isn’t Lambda in the definitions of f/g/kappa??”. Anyway it makes perfect sense when you define it clearly as after Eqn 18.
e) At the beginning of Section 3.2 you have a clause “the over-shoot gates act more like on the output”, which doesn’t make sense.
f) Figures should be approximately self-explanatory—but “Ratio of lower components” is not explained in the caption of Fig 4b. Although I greatly appreciate that you have made the actual plot lines/markers very easy to see! Should these results be averaged over many training attempts with error bars?


Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint-level annotations. A popular approach is to factorize an image into a pose and appearance data stream, then to reconstruct the image from the factorized components. The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks to focus on the foreground object of interest. However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background. This work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions, conditioning only the foreground reconstruction on the unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest. Furthermore, the rendered background quality is also improved, as the background rendering pipeline no longer requires the ill-suited landmarks to model its pose and appearance. We demonstrate this improvement in the context of the video-prediction task.

The paper presents an unsupervised approach for learning landmarks in images or videos with single objects by separating the representation of the image into foreground and background and factorizing the representation of the foreground into pose and appearance. It builds upon previous work [Jakab 2018, Lorenz 2019] who proposed to train by reconstructing the original image from one version of the image with perturbed appearance and another with perturbed pose. It extends this approach by introducing an additional separation of foreground and background in the image.
Strengths:
+ Nicely motivates the approach of separating foreground and background
+ Fewer landmarks are needed than in previous work
+ Approach seems beneficial for video prediction
+ Clear and well written
+ Detailed description of architecture and training

Weaknesses:
- The changes and improvements feel somewhat incremental
- Some uncertainty about the solidity of the evaluation/comparability with baselines
Results on CelebA somewhat weak
Overall the paper is well written, easy to follow, presents a straightforward extension of previous work and appears to show an improvement. I’m thus generally supportive of the paper.
One question I’d like to see addressed in the response is about the evaluation: as you state, the details about cropping are not known for previous work, introducing some uncertainty into the external comparisons. It would be great if you could provide some more details about the steps you took to verify that your internal baseline is indeed comparable to previous work (e.g. Lorenz 2019). For instance, on CelebA your baseline seems to fall short of Lorenz 2019, which may suggest that your substantial looking improvement on BBC Pose is indeed due to more favorable cropping.
The second concern is about the results on CelebA presented in Fig. 6 in the Appendix: It looks like the background net reconstructs almost the entire image. I would have expected that hair style, shape of ears or existence of a beard would equally warrant landmarks and appearance. It seems a bit odd that the foreground is so focused on the central part of the face. Do you have an explanation for that?
Minor comments:
- Test in Fig. 1 too small and not readable when printed
- Fig. 1 seems to be missing an arrow pointing from image to appearance encoder
- Why are there more than 8 points in Fig. 2 for “Ours8” (and more than 12 for “Ours12”)?


Weight-sharing-the simultaneous optimization of multiple neural networks using the same parameters-has emerged as a key component of state-of-the-art neural architecture search. However, its success is poorly understood and often found to be surprising. We argue that, rather than just being an optimization trick, the weight-sharing approach is induced by the relaxation of a structured hypothesis space, and introduces new algorithmic and theoretical challenges as well as applications beyond neural architecture search. Algorithmically, we show how the geometry of ERM for weight-sharing requires greater care when designing gradientbased minimization methods and apply tools from non-convex non-Euclidean optimization to give general-purpose algorithms that adapt to the underlying structure. We further analyze the learning-theoretic behavior of the bilevel optimization solved by practical weight-sharing methods. Next, using kernel configuration and NLP feature selection as case studies, we demonstrate how weight-sharing applies to the architecture search generalization of NAS and effectively optimizes the resulting bilevel objective. Finally, we use our optimization analysis to develop a simple exponentiated gradient method for NAS that aligns with the underlying optimization geometry and matches state-of-the-art approaches on CIFAR-10.

I have not worked in the optimization filed and I am only gently followed the NAS field. I might under-valued the theoretical contribution.
This work provides  theoretical analysis for the NAS using weight sharing in two aspects:
1) The authors give non-asymptotic stationary-point convergence guarantees (based on stochastic block mirror descent (SBMD) from Dang and Lan (2015)) for the empirical risk minimization (ERM) objective associated with weight-sharing. Based on this analysis, the authors proposed to use  exponentiated gradient to update architecture parameter, which enjoys faster convergence rate than the original results in Dang and Lan (2015). The author also provided an alternative to SBMD that uses alternating successive convex approximation (ASCA) which has similar convergence rate.
2) The author provide generalization guarantees for this objective over structured hypothesis spaces associated with a finite set of architectures.
My biggest concern is the validity of the proposed exponentiated gradient update, at least empirically. We indeed observed slightly improvement in test error over DARTS on the CIFAR10 benchmark but how reproducible the results are? Can you compare at least on the other benchmark (PENN TREEBANK) used in Liu et al 2019? Also, comparing to first order DARTS, search cost is the same and this is hard to justify the better convergence rate for EDARTS. In addition, the results on feature map selection is not very encouraging as the gap to the successive halving is significant.
The author proposed ASCA, as an alternative method to SBMD. Why we need such alternative? What is the advantage of ASCA comparing to SBMD? When should I use ASCA and when SBMD? How do they empirically different?
Then I feel some wording can be improved. For example, "while requiring computation training …”,  “…which may be of independent interest”.


Traditional set prediction models can struggle with simple datasets due to an issue we call the responsibility problem. We introduce a pooling method for sets of feature vectors based on sorting features across elements of the set. This can be used to construct a permutation-equivariant auto-encoder that avoids this responsibility problem. On a toy dataset of polygons and a set version of MNIST, we show that such an auto-encoder produces considerably better reconstructions and representations. Replacing the pooling function in existing set encoders with FSPool improves accuracy and convergence speed on a variety of datasets.
Decoder: This turns the latent space back into a set. The elements in the target set have an arbitrary order, so a standard reconstruction loss cannot be used naïvely -the decoder would have to somehow output the elements in the same arbitrary order. Methods like those in Achlioptas et al. (2018) therefore use an assignment mechanism to match up elements (section 2), after which a usual reconstruction loss can be computed. Surprisingly, their model is still unable to solve the polygon reconstruction task with close-to-zero reconstruction error, despite the apparent simplicity of the dataset.
In this paper, we introduce a set pooling method for neural networks that addresses both the encoding bottleneck issue and the decoding failure issue. We make the following contributions:
1. We identify the responsibility problem (section 3). This is a fundamental issue with existing set prediction models that has not been considered in the literature before, explaining why these models struggle to model even the simple polygon dataset. 2. We introduce FSPOOL: a differentiable, sorting-based pooling method for variable-size sets (section 4). By using our pooling in the encoder of a set auto-encoder and inverting the sorting in the decoder, we can train it with the usual MSE loss for reconstruction without the need for an assignment-based loss. This avoids the responsibility problem. 3. We show that our auto-encoder can learn polygon reconstructions with close-to-zero error, which is not possible with existing set auto-encoders (subsection 6.1). This benefit transfers over to a set version of MNIST, where the quality of reconstruction and learned representation is improved (subsection 6.2). In further classification experiments on CLEVR (subsection 6.3) and several graph classification datasets (subsection 6.4), using FSPool in a set encoder improves over many non-trivial baselines. Lastly, we show that combining FSPool with Relation Networks significantly improves over standard Relation Networks in a model that heavily relies on the quality of the representation (subsection 6.5).

This paper proposes to make permutation-equivariant set autoencoders, by introducing a permutation invariant encoder (based on feature-wise sorting) and at the same time an 'inverse' operation (that undoes the sorting) in the decoder. The achieved equivariance allows the autoencoders to be trained end-2-end without a matching-based loss (like Chamfer) which I think is a very neat result. Also, to address potentially variable-sized sets as inputs/outputs, the authors propose to use a calibrator function that effectively 'samples' the produced features on the set's elements in fixed intervals, that are auto-adaptable in the (variable) set size.
The results shown are well placed in the current literature and form an important novelty that should have non-trivial impact in the sub-field. Also, the experiments done are well executed, with ample variability in the nature of the datasets used and I expect them to also be easily reproducible (given the authors' provided code).
Some points I would appreciate to see an improvement:
a. The intro is rough (please see minor-specifics at places that you can improve the exposition).
b. The Figure1 is not easy to read. The 90degree rotation results in the same set, but as a 'pictorial' image, obviously not.
c. The "responsibility problem" is already very well explained in the Zhang19a. I would appreciate to tone-down in this paper, the "discovery" of it as a main contribution.
d. Experiments 6.2. it appears that the FSPool/unpool model is better *only* when the mask features are been considered. What are these mask-features? Why do they matter?
e. The way you describe the responsibility problem (discontinuity) is very hand-wavy. It would be nice to explicitly it write it in rigorous math.
f. Why using the relaxation of Grover et al. helped you to avoid the discontinuity that would be otherwise introduced via standard sorting? (I am not familiar with their exact relaxation, but intuitively, their method been a good proxy for sorting, should suffer from it as well).
Explicit Minor Comments on writing:
(all in introduction)
-4rth line: "this" -> this problem
-"Methods like by" -> Methods like those in
-"In this paper, we introduce a set pooling method for neural networks that addresses both issues" -> which issues? the encoder's collapse and the decoder's inneficiency in matching? Please explain.
-"good baselines" -> sophisticated/non-trivial baselines
Appendix. Table 9, max-pool at \sigma=0, seems to be the best (please use boldface to indicate it).


This work presents a modular and hierarchical approach to learn policies for exploring 3D environments. Our approach leverages the strengths of both classical and learning-based methods, by using analytical path planners with learned mappers, and global and local policies. Use of learning provides flexibility with respect to input modalities (in mapper), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies). Such use of learning within each module retains its benefits, while at the same time, hierarchical decomposition and modular training allow us to sidestep the high sample complexities associated with training end-to-end policies. Our experiments in visually and physically realistic simulated 3D environments demonstrate the effectiveness of our approach over past learning and geometry-based approaches. 1 1 INTRODUCTION Navigation is a critical task in building intelligent agents. Navigation tasks can be expressed in many forms, for example, point goal tasks involve navigating to a specific coordinates and semantic navigation involves finding path to a specific scene or object. Such tasks may need to be performed in known (already mapped) or unknown environments. Irrespective of the task or the setting, a core problem in navigation is exploration, i.e., how to efficiently visit as much of the environment. This is useful for pre-mapping in known environments, or actually solving tasks in known environments.
Recent work from  has used end-to-end learning to tackle this problem. Their motivation is three fold: a) learning provides flexibility to the choice of input modalities (classical systems rely on observing geometry through use of specialized sensors, while learning systems can infer geometry directly from RGB images), b) use of learning can improve robustness to errors in explicit state estimation, and c) learning can effectively leverage structural regularities of the real world, leading to more efficient behavior in previously unseen environments. This lead to their design of an end-to-end trained neural network based policy that processed raw sensory observations to directly output actions that the agent should execute.
While use of learning for exploration is well motivated, casting the exploration problem as an end-to-end learning problem has its own drawbacks. Learning about mapping, state-estimation and path-planning purely from data in an end-to-end manner can be prohibitively expensive. Consequently, past end-to-end learning work for exploration from  relies on use of imitation learning and many millions of frames of experience, but still performs worse than classical methods that don't require any training at all. This motivates our work. In this paper, we investigate alternate formulations of employing learning for exploration that retains the advantages that learning has to offer, but doesn't suffer from the drawbacks of full-blown end-to-end learning. Our key conceptual insight is that use of learning for leveraging structural regularities of indoor environments, robustness to state-estimation errors, and flexibility with respect to input modalities, happens at different time scales and can thus be factored out. This motivates use of learning in a modular and hierarchical fashion inside of what one may call a 'classical navigation pipeline'. This results in navigation policies that can work with raw sensory inputs such as RGB images, are robust to state estimation errors, and leverage regularities of real 1 See https://sites.google.com/view/active-neural-mapping for visualization videos.

The paper describes a method for visual robot navigation in simulated environments. In terms of overall objectives and targeted reasoning, the current approaches can be roughly divided into two groups: i) learning tasks requiring high-level reasoning for navigation involving the detection and discovery of objects and their affordances and eventually also requiring to process language input, and ii) simpler navigation task involving geometry and the detection of free (navigable space): point goal, maximizing coverage etc. The former target more complex problems but the agents are more difficult (currently up to impossible) to transfer to real environments, whereas the latter directly target problems which can currently realistically used in real world scenarios.
The paper is of the second group, and addresses one of the currently investigated problems in robot navigation and mapping, namely whether learned navigation is superior to traditional planning algorithms, and whether the two different approaches can be integrated. It proposes to separate the task into long-term and short-term goals, which is not new per se, but the proposed formulation is quite interesting. In particular, the integration of the “handcrafted” planar (front propagation) into the learned framework solves a couple of issues with sample efficiency of learned methods, while still keeping some flexibility of learning over the 100% traditional approaches.
I will be upfront – I already reviewed an earlier version of this paper for NeurIPS 2019, where this paper unfortunately did not pass. I was actually a favorable reviewer at this time and was defending it. The paper has been improved since and I would be happy to see it pass. I still have a couple of questions, some of which are similar to the ones I raised in the NeurIPS review (others have been addressed since).
While I do agree that the targeted tasks might be considered less exciting then tasks involving high level semantics, I do also think that these tasks are far from solved as soon as we try to implement them in real life scenarios. I do think that the proposed paper is an interesting step forward.
The advantages of the proposed method are “bought” with a couple of key design choices, in particular the handcrafted non-differentiable long-term path planner. The downside of this is that the loss signals can’t be backpropagated through the planner, which restricts the mapping module to very simple mapping information, basically free /navigational space. End-to-end training of navigation could in principle learn to map objects and affordances which are discovered through the task and not hardcoded or even learned with supervision, which also must be known in advance. This means that the contribution is limited to simpler navigational tasks like the tested exploration and PointGoal. In contrast, other work from the literature uses differentiable planners (eg cited CMP (Gupta et al 2017), using value Iteration Networks (cited Talmar et al. 2016) which allows fine-tuning.
The mapping network, which is learned with supervision, is a general encoder-decoder network which needs to translate from projective first person views to ego-centric bird’s eye views. It thus needs to learn projective geometry from data, although projective geometry could be used as structure for the network, given camera calibration, which has been done in other work:
-	Chen et al., 2019
-	Gupta et la., 2017
-	Henriques et al., 2018
And a couple of others.
Several improvements have been made since the NeurIPS submission, some of which I had addressed in my review. The experiments are quite convincing in their comparisons with the state of the art, in particular the generalization performances:
- generalization from Gibson (training) to Matterport (testing)
- generalization from exploration (training) to PointGoal (testing).
A couple of the results have been removed from the NeurIPS submission, unfortunately, I think they should be kept in.
I appreciated the realistic sensor model fitted to real data measured with a Locobot robot, and the ablation studies, which indicated the contributions of the different planner modules and of pose estimation. The role of the short term planner has been made clearer in the new paper.
I found it interesting that the stellar performance at the Habitat AI challenge was removed from the new paper – this method (or at least a preceding version) won the challenge. But I do understand that this choice was motivated by some remarks of the NeurIPS fellow reviewers regarding the simplicity of the PointGoal task of the challenge.
A couple of less positive aspects, and questions:
On the downside, and following the remarks on literature above, I still think that the results should be compared with CMP, the main competitor of this method.
I think this is the main short coming of the paper, in particular since CMP is able to perform end-to-end training because the planner is differentiable (value iteration networks, NIPS 2016).
The literature w.r.t. to hierarchical planning is very far from exhaustive and lots of work is missing, consisting of recent work
Embodied Question Answering, Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra, CVPR 2018
(and several follow up papers)
but also quite classical work like the literature around the options framework, with the following starting point:
R.S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181–211, 1999.
And many other papers.
The figures 1 and 2 have been completely redone, but they are not completely clear. In particular, several intermediate representations/maps/Images are not commented or labeled, they should be annotated with the symbols from the text.
The role of the sensor output is not clear. Sensors normally provide relative positions … but the text seems to indicate absolute pose. Some details are lacking.
In “… to predict the pose change between the two maps …” it is unclear what is done here. Is this self-supervision?
The authors mention that unexplored area is considered as free space for planning. What consequences did this have in case of unexplored obstacles? I guess the problem was delegated to the local policy, which needed coping with these issues?
The last paragraph before the conclusions briefly mentions experiments and comparisons but without giving any details. This is unfortunate, since there is still space available (the paper length is 8.5 pages).


Predictive coding theories suggest that the brain learns by predicting observations at various levels of abstraction. One of the most basic prediction tasks is view prediction: how would a given scene look from an alternative viewpoint? Humans excel at this task. Our ability to imagine and fill in missing visual information is tightly coupled with perception: we feel as if we see the world in 3 dimensions, while in fact, information from only the front surface of the world hits our (2D) retinas. This paper explores the connection between view-predictive representation learning and its role in the development of 3D visual recognition. We propose inverse graphics networks, which take as input 2.5D video streams captured by a moving camera, and map to stable 3D feature maps of the scene, by disentangling the scene content from the motion of the camera. The model can also project its 3D feature maps to novel viewpoints, to predict and match against target views. We propose contrastive prediction losses that can handle stochasticity of the visual input and can scale view-predictive learning to more photorealistic scenes than those considered in previous works. We show that the proposed model learns 3D visual representations useful for (1) semi-supervised learning of 3D object detectors, and (2) unsupervised learning of 3D moving object detectors, by estimating motion of the inferred 3D feature maps in videos of dynamic scenes. To the best of our knowledge, this is the first work that empirically shows view prediction to be a useful and scalable self-supervised task beneficial to 3D object detection.

This paper deals with turning a 2.5D video representation into a 3D representation of an environment or a scene. The authors introduce self-supervised methods to pretrain the 2d-3d projection with a contrastive loss, which is the neural backbone for multiple other tasks. They then assess their approach on numerous tasks such as 3D-object detection, 3D-moving object detection, and 3D motion estimation. The authors also evaluate the transferability of the features in a challenging sim2real setting.
It is dense paper with multiple modules (2d-3d, ergomotion, memory, etc.) and concept. Still, the authors make it accessible by concise paragraphs, highlighting key equations (The enum + eq 1 and 2 are quite useful), and well-designed sketch (Figure1). I had some difficulties digging into the visual head component for each task as I was not familiar with this topic. However, the authors always explain their choices in a few lines and refer to the related papers for technical details in a meaningful way.
I am pretty convinced with the experiments, especially Sim2Real, in Tab1, where the baselines are clears and make sense.
I appreciated the limitation section, which is transparent and honest, and clearly states the strength and weaknesses (such as image downscaling) of the approach. Besides, the code and the data should be released, which is always a positive point.
Remarks:
 - Latent map update: running average is a simple and efficient mechanism, it also makes sense as you are dealing with big 3D tensors. Yet, have you tried other update mechanisms?
 - A natural follow-up to this paper is Contrastive Predictive Losses (which had several successes in pure vision setting[1]). Did you already try this approach?
 - In visual CPC papers [1] (or since the early days of visual representation learning!), data transformation has been applied to improve model performance. Would it make sense to apply it to I_{n+1}, D_{n+1} ?
 - Although the authors assess their approach with RGB-D, the models were still trained on 2.5D video. It would have been useful also to assess a pretraining on pure RGB-D data
However, I have two (somehow related) concerns. First of all, the machine learning novelties are rather small, contrastive losses are now widespread, and the models are closed to Tung et al. as mentioned by the authors.  However, I believe the paper to be a substantial contribution in vision, as they show the feasibility of their approach on large scale scenarios and over a highly diverse set of tasks. Again, the authors also release the code, making the paper a valuable baseline for the following work. On my side, I am impressed by the sim2real env.
My second concern is the following, it is a high quality vision paper, and I am curious why the authors chose ICLR over CVPR. Besides, the tasks are vision-oriented, and 2.5D vision is not common in the ML community. Having said that, the proposed approach is pretty generic, can be applied to RGB-D (more common in ML), and require few expert knowledge in vision (only the Egomotion module).
As a result, I would advocate for clear accept if we assess vision-based contribution for ICLR; otherwise, I would only recommend weak accept the paper is solely based on ML contributions (the paper is still sound, well-written, with numerous experiments and with a semi-generic architecture)


We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (FPrinciple) -DNNs often fit target functions from low to high frequencies -on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With theories under an idealized setting, we illustrate that this F-Principle results from the smoothness/regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or a randomized dataset.

The paper studies the training process of NNs through the lens of Fourier analysis. The authors argue that during the training process, NNs will first learn low frequencies part of the function first and then the high frequency part. To verify this claim empirically, the author propose two methods: 1. examine the convergence of different frequencies in a pre-selected direction in the frequency space during training; 2. examine the convergence rate of the 2-norm of low v.s. high frequencies during training.  Through the experimental results of these two methods, the authors conclude that NNs learn the low  frequency components before the high frequency components. The authors also discuss a potential application of this observation to solving high dimensional PDEs: coupling DNNs training (good at learning low frequency components) with the Jacobi method (good at learning high frequency components). Finally, the authors also provide some theoretical intuition (Thm 1., 2.) why low frequency components are learned faster and an explanation why NNs could generalize well on images but perform poorly on tasks like learning parity functions.
Other comments:
1. It seems the filtering method is a better (might be a sufficient) way to justify the F-Principle than the projection method, given the projection method examines only one direction (also appointed out in the paper).
2. When talking about Fourier transform, would you specify what is the domain of the functions and how the functions are defined (section 3.1) The notation there is somewhat confusing (which makes the rest of the paper difficult to follow) since you are mentioning the Fourier transform of the set {(x_i, y_i)}. It will be helpful to define the function before defining its Fourier transform.  Please also mention what is the domain of the function, {x_i}_i or R^d?
3. According to equation (4), it seems the domain of the functions is {x_i}_i, otherwise equation (4) should be a function of x\in R^d, not x_i.
4. Could you elaborate why (4) is a good approximation of the low frequency energy rather than the L2 norm (over x\in R^d) of (4) with x_i replaced by x\in R^d.
5. It might be useful to refine the related work section. It is not clear what are the previous contributions prior to this paper, and it seems [1] shares some similar results/observation with the this paper.
Overall, I lean to a weak rejection. The key findings (and similar results, e.g. NNs learn simple functions first), i.e. F-Principle seems to have already appeared in previous works, e.g. [1] and the theoretical results of this paper are limited to an idealized setting (results of more general setting appear in another work, mentioned in the paper.)
[1]Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht,
Yoshua Bengio, and Aaron Courville. On the spectral bias of deep neural networks. arXiv preprint
arXiv:1806.08734, 2018. 1, 8, A


Parameter quantization is a popular model compression technique due to its regular form and high compression ratio. In particular, quantization based on binary codes is gaining attention because each quantized bit can be directly utilized for computations without dequantization using look-up tables. Previous attempts, however, only allow for integer numbers of quantization bits, which ends up restricting the search space for compression ratio and accuracy. Moreover, quantization bits are usually obtained by minimizing quantization loss in a local manner that does not directly correspond to minimizing the loss function. In this paper, we propose an encryption algorithm/architecture to compress quantized weights in order to achieve fractional numbers of bits per weight and new compression configurations further optimize accuracy/compression trade-offs. Decryption is implemented using XOR gates added into the neural network model and described as tanh(x), which enable gradient calculations superior to the straight-through gradient method. We perform experiments using MNIST, CIFAR-10, and ImageNet to show that inserting XOR gates learns quantization/encrypted bit decisions through training and obtains high accuracy even for fractional sub 1-bit weights.

Summary:
The authors propose quantize the weights of a neural network by enabling a fractional number of bits per weight. They use a network of differentiable XOR gates that maps encrypted weights to higher-dimensional decrypted weights to decode the parameters on-the-fly and learn both the encrypted weights and the scaling factors involved in the XOR networks by gradient descent.
Strengths of the paper:
- The method allows for a fractional number of bits per weights and relies of well-known differentiable approximations of the sign function. Indeed, virtually any number of bits/weights can be attained by varying the ratio N_in/N_out.
- The papers displays good results on ImageNet for a ResNet-18.
Weaknesses of the paper:
- Some arguments that are presented could deserve a bit more precision. For instance, quantizing to a fractional number of bits per weights per layer is in itself interesting. However, if we were to quantize different layers of the same network with distinct integer  ratio of bits per weights (say 1 bit per weight for some particular layers and 2 bits per weight for the other layers), the average ratio would also be fractional (see for instance "Hardware-aware Automated Quantization with Mixed Precision", Wang et al., where the authors find the right (integer) number of bits/weights per layer using RL). Similarly, using vector quantization does allow for on-chip low memory: we do not need to re-instantiate the compressed layer but we can compute the forward in the compressed domain (by splitting the activations into similar block sizes and computing dot products).
- More extensive and thorough experiments could improve the impact of the paper. For instance, authors could compress the widely used (and more challenging) ResNet-50 architecture, or try other tasks such as image detection (Mask R-CNN). The table is missing results from: "Hardware Automated Quantization", Wang et al ; "Trained Ternary Quantization", Zhu et al ; "Deep Compression",  Han et al; "Ternary weight networks", Li et al (not an extensive list).
- Similarly, providing some code and numbers for inference time would greatly strengthen the paper and the possible usage of this method by the community. Indeed, I wonder what the overhead of decrypting the weights on-the-fly is (although it only involves XOR operations and products)
- Small typos: for instance, two points at the very end of section 5.
Justification fo rating:
The proposed method is well presented and illustrated. However, I think the paper would need either (1) more thorough experimental results (see comments above, points 2 and 3 of weaknesses) or (2) more justifications for its existence (see comments above, point 1 of weaknesses).


We study the problem of Reinforcement learning from demonstrations (RLfD), where the learner is provided with both some expert demonstrations and reinforcement signals from the environment. One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations. Another approach uses demonstration data for reward shaping. By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Specifically, such reward shaping approach trains an agent not only to imitate demonstrated actions when it encounters demonstrated states, but also to reach demonstrates states, when it confronts states that are not observed in the demonstration data. However, existing algorithms in the latter one adopt shaping reward which is not directly dependent on current policy, limiting the algorithms to treat demonstrated states the same as other states, and fail to directly exploit supervision signal in demonstration data. In this paper, we propose a novel objective function with policy-dependent shaping reward, so as to get the best of both worlds. We present a convergence proof for policy iteration of the proposed objective, under the tabular setting. Then we develop a new practical algorithm, termed as Demonstration Actor Critic (DAC). Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a significant performance gain over five strong and off-the-shelf baselines.

This paper presents a method for doing RL from demonstrations in continuous control tasks. It combines both an augmented reward for minimizing the KL between the policy and the expert actions as well as directly minimizing that KL in the policy. Results on 5 sparse reward mujoco tasks show that it out-performs other related methods.
The motivation for the paper is difficult to follow. They claim that using demonstration data in a supervised manner "cannot generalize supervision signal over those states unseen in the demonstrations," but most of these approaches are using neural networks and definitely are generalizing those signals to other states. Whether they're generalizing accurately or not is a different question. In contrast, they say that reward shaping approaches do not suffer that problem because they evaluate trajectories rather than states, but there will still be a problem of generalizing to new trajectories.
The abstract is even more confusing as it tries to jump straight into the issues with these approaches without any explanation. I don't think there's enough space in the abstract to go into that level of detail.
The description of DQfD and DDPGfD in the related work is not accurate. They're described as "treating demonstration data as self-generated data," but in fact they both add supervised losses to more closely match the demonstrated data. https://ieeexplore.ieee.org/document/8794074 is another method built on DDPG that has both a critic and actor loss like yours and would make a useful comparison.
The related work section should also discuss and compare/contrast to GAIL, I was surprised that wasn't in there, especially since you also use a discriminator to differentiate expert and agent actions.
The end of the related work section is not very clear, you say these methods are problematic because "the adopted shaping reward yields no direct dependence on the current policy" but there's no explanation or motivation for why that would be a problem.
Assumption 1 seems like a very strong assumption that would not be true for many human experts.
For the experiments, I wonder about the impact of only using sparse reward tasks. Converting the tasks to sparse reward in this way makes them partially observable, and then potentially the expert demonstrations are required to overcome that partial observability. How do the methods compare on the unmodified tasks? There was nothing specific in your algorithm that meant it should specifically address sparse reward tasks. What about tasks that are naturally sparse reward?
Overall, the algorithm is interesting and the results are nice. The motivation and related work need to be made clearer to situate this work with the other related works. And the experiments should go beyond these tasks that have been modified to have sparse rewards.
The revised version of the paper addresses many of my concerns about the motivation, related works, and comparisons with GAIL, so I'm updating my score to Weak Accept.