We present a novel architecture of GAN for a disentangled representation learning. The new model architecture is inspired by Information Bottleneck (IB) theory thereby named IB-GAN. IB-GAN objective is similar to that of InfoGAN but has a crucial difference; a capacity regularization for mutual information is adopted, thanks to which the generator of IB-GAN can harness a latent representation in disentangled and interpretable manner. To facilitate the optimization of IB-GAN in practice, a new variational upper-bound is derived. With experiments on CelebA, 3DChairs, and dSprites datasets, we demonstrate that the visual quality of samples generated by IB-GAN is often better than those by β-VAEs. Moreover, IB-GAN achieves much higher disentanglement metrics score than β-VAEs or InfoGAN on the dSprites dataset.

(Apologies for this belated review)
Summary
The authors propose a GAN-based approach to learning disentangled representations that combines elements InfoGAN with recent Information-Bottleneck (IB) perspectives on variational auto-encoders. In addition to minimizing the normal GAN loss, the authors propose to maximize a lower bound on the mutual information under the generative model, whilst minimizing an upper bound
	Ig[X,Z] = E_p(X,Z)[log p(X,Z) - log p(X) - log p(Z)]
In order to optimize this objective whilst retaining the likelihood-free property of GANs, the authors propose to define a generative model with an intermediate representation r, which allows them to define a likelihood-free generator x = G(r) whilst defining a parametric distribution p(r,z) = eψ(r|z) p(z). This enables the authors to define model architectures that jointly train an encoder qφ(z | x) and a GAN-style generator using an objective that incorporates inductive biases for learning disentangled representations
Quantitative evaluation is performed on d-Sprites (where metrics for disentanglement are evaluated), and qualitative results are shown for Celeb-A and the Chairs dataset.
Comments
I think this is a paper that presents several interesting ideas. Integrating IB-based ideas into the InfoGAN framework is a useful contribution. Moreover, I think that the way they authors integrate a likelihood-free generative model with an inference model is something of a contribution in its own right. I particularly like the idea of the intermediate representation.
Having done some work in this space, I would say that the results on d-Sprites are quite good. Aside from the numerical scores in the table aside, the latent traversals in Figure 2 show a good degree of disentanglement. There is a reason that many of the recent papers don’t show these traversals; it turns out to quite difficult to disentangle shape from the other variables, and even rotation tends to correlate with some of the other latents in many cases.
That said, I would say that the experiments could do with some additional work. I would like to see some discussion of how tight/loose the upper and lower bounds are (some convergence plots would be helpful in this regard). I would also like to see some experiments that evaluate different choices for λ and β (along with some discussion of how these values were chosen – see below). Finally, could the authors find one or two additional datasets? I generally find it difficult to evaluate results on Celeb-A (other than the qualitative evaluation “the images look sharper than those produced by VAEs”). Even something like MNIST/FMNIST would be OK for purposes of evaluating inclusion of Discrete/Concrete variables and/or extrapolation to unseen combinations of factors (as in the Esmaeli et al. paper).
Overall, I would say that this is a potentially strong paper, but that experimental evaluation does need work. I’d be willing to look at an updated version of the paper and adjust my score accordingly if the authors can provide one.
Questions
- Could the authors comment on why they need to set λ=150, β=1? On a quick read, it is not immediately obvious to me why λ > β implies that we will maximize Ig[X,Z] is this simply because maximizing the lower bound will win out over minimizing the upper bound? When we set λ=β, since this would yield a zero loss when the bounds are tight, is that correct? In this case we presumably not necessarily expect to maximize Ig[X,Z] w.r.t. θ?
- What is perhaps missing from this paper is a discussion of *why* maximizing Ig[X,Z] induces a disentangled representation. One hypothesis could be that be that, for a given number of uncorrelated latent variables, a disentangled generator is simply more efficient in terms of the number of distinct samples X it can construct. In this context, it would be interesting if the the authors could report their Ig[X,Z] bounds. In particular, could they compute
	exp[Ig[X,Z]] / N

Intuitively, this number indicates how many examples the generator can produce relative to the number of training examples N. Is it the case that a more disentangled generator is also capable of producing more distinct samples?
Minor
- This is a bit of a pet peeve of mine: Is it really true that GANs lean a representation? A representation is generally a mapping from data to features. A GAN is a mapping from features to data. The authors in this paper do train an encoder to invert the generative model, which learns representation, and certainly a disentangled GAN arguably is useful for more controllable forms of generation in its own right, it just seems that we should not conflate the two.
- Fix: General consent -> General consensus
- Fix: good representation -> good representations
- Fix: such disentangled representation -> disentangled representations
- Fix: (?Higgins et al., 2017b; 2018)


Designing a molecule with desired properties is one of the biggest challenges in drug development, as it requires optimization of chemical compound structures with respect to many complex properties. To augment the compound design process we introduce Mol-CycleGAN -a CycleGAN-based model that generates optimized compounds with a chemical scaffold of interest. Namely, given a molecule our model generates a structurally similar one with an optimized value of the considered property. We evaluate the performance of the model on selected optimization objectives related to structural properties (presence of halogen groups, number of aromatic rings) and to a physicochemical property (penalized logP). In the task of optimization of penalized logP of drug-like molecules our model significantly outperforms previous results.

The paper presents an approach for optimizing molecular properties, based on the application of CycleGANs to variational autoencoders for molecules. A recently proposed domain-specific VAE called Junction Tree VAE (JT-VAE) is employed. The optimization of molecules is an important problem for example in drug discovery or materials design, and the development of machine learning approaches for it is therefore an important application.
On the positive side, this reviewer would like to highlight the structural transformation task, which is an interesting addition. The presentation is mostly clear, and there is an improvement in benchmark scores compared to the baseline JT-VAE.
On the negative side, this reviewer would argue the paper is a bit incremental, as it appears to be “just” a combination of two models without a clear motivation from a molecular design perspective – why is the model e.g. better (motivated) than the model by You et al, the MOLGAN by DeCao and Kipf, or the CGVAE by Liu et al?
 Also, the number of benchmark experiments is small compared to more rigorous evaluation studies, e.g. by Olivecrona et al. https://doi.org/10.1186/s13321-017-0235-x
Furthermore, a lot of (labeled) training examples for the respective properties seem to be needed, whereas in practical drug discovery, often only very small datasets are available.
Questions:
How exactly is the similarity constraint enforced?
Why do the authors use the JT-VAE and not the SD-VAE or CGVAE?
Can the authors explain why they think penalized logP is relevant to drug discovery? While the optimization of physicochemical properties alone is certainly important in a multi-objective setting, this reviewer is not entirely sure that physicochemical property optimization alone is a problem that often occurs in drug discovery.
Why are the values for the RL model by You et al. 2018 not shown in Table 2, even though the authors state that they have done the comparison? If those molecules are not drug-like, it would be good to see the comparison, and show the respective molecules. Keep in mind that you are not asking the models to produce drug-like molecules here! The RL model therefore seems to generate exactly what it is asked (rewarded) for. Is it a fair comparison that you expect it to do something else?
Does the requirement to define the sets X and Y restrict the capacity of the model to binary class membership, as in “good” and “bad”? Does this imply the model cannot extrapolate into e.g. continuous parameter ranges unseen during training?
Provocative statement this reviewer would be keen to hear the authors’ (and co-reviewers) opinion on: It seems that there are considerable efforts needed in getting VAEs for discrete data such as graphs to work, namely the drastic restriction of the molecular graph space via grammars, CGVAE, or the JT setting (which builds on a heuristically defined vocabulary), whereas RL models even get decent results with simple SMILES-RNNs (see e.g. the work by Merk et al, who have even successfully made and tested their results in the lab!  https://doi.org/10.1038/s42004-018-0068-1 and https://doi.org/10.1002/minf.201700153 ) Do the author have an intuition into why this might be the case?
Regarding the proposed application of GANs to text generation, this reviewer would suggest to have a look at this recent (as in yesterday) paper https://arxiv.org/pdf/1811.02549.pdf
Overall, this reviewer believes the paper would be a good ICLR workshop paper or, after a bit more technical details on the models, a decent chemoinformatics paper, but unfortunately this reviewer is not convinced that in the current form the results are significant or general enough in terms of ML to warrant acceptance in the main ICLR track.


Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution -that of using experience replay buffers for all past events -with a mixture of on-and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

This paper proposes a particular variant of experience replay with behavior cloning as a method for continual learning. The approach achieves good performance while not requiring a task label. This paper makes the point that I definitely agree with that all of the approaches being considered should compare to experience replay and that in reality many of them rarely do better. However, I am not totally convinced when it comes to the value of the actual novel aspects of this paper.
Much of the empirical analysis of experience replay (i.e. the buffer size, the ratio of past and novel experiences, etc…) was not surprising or particular novel in my eyes. The idea of using behavior cloning is motivated fully through the lens of catastrophic forgetting and promoting stability and does not at all address achieving plasticity. This was interesting to me as the authors do mention the stability-plasticity dilemma, but a more theoretical analysis of why behavior cloning is somehow the right method among various choices to promote stability while not sacrificing or improving plasticity was definitely missing for me. Other options can certainly be considered as well if your aim is just to add stability to experience replay such a notion of weight importance for the past like in EwC (Kirkpatric et al., 2017) and many other papers or using knowledge distillation like LwF (Li and Hoeim, 2016). LwF in particular seems quite related. I wonder how LwF + experience replay compares to the approach proposed here. In general the discourse could become a lot strong in my eyes if it really considered various alternatives and explained why behavior cloning provides theoretical value.
Overall, behavior cloning seems to help a little bit based on the experiments provided, but this finding is very likely indicative of the particular problem setting and seemingly not really a game changer. In the paper, they explore settings with fairly prolonged periods of training in each RL domain one at a time. If the problem was to become more non-stationary with more frequent switching (i.e. more in line with the motivation of lifelong learning), I would imagine that increasing stability is not necessarily a good thing and may slow down future learning.


Few-Shot Learning (learning with limited labeled data) aims to overcome the limitations of traditional machine learning approaches which require thousands of labeled examples to train an effective model. Considered as a hallmark of human intelligence, the community has recently witnessed several contributions on this topic, in particular through meta-learning, where a model learns how to learn an effective model for few-shot learning. The main idea is to acquire prior knowledge from a set of training tasks, which is then used to perform (few-shot) test tasks. Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks. This is a very strong assumption which restricts the usage of meta-learning strategies in the real world where ample training tasks following the same distribution as test tasks may not be available. In this paper, we propose a novel meta-learning paradigm wherein a few-shot learning model is learnt, which simultaneously overcomes domain shift between the train and test tasks via adversarial domain adaptation. We demonstrate the efficacy the proposed method through extensive experiments.

This paper proposes to combine unsupervised adversarial domain adaptation with prototypical networks and finds that the proposed model performs well on few-shot learning task with domain shift, much better than other few-shot learning baselines that do not consider. Specifically it tests on Omniglot with natural image background and cliparts to real images.
It is true that current meta-learning approaches do not address the problem of domain shift, and as a result, the testing domain has to be the same with the training domain. However, this paper rather than proposing solution address the meta-learning problem, albeit the title “meta domain adaptation”, only brings few-shot learning to domain adaptation. Here’s why:
In order for a meta-learning model to be called “meta domain adaptation,” the type of adaptation cannot be seen during training, and the goal is to test on adaptation that the model has not seen before. Indeed, each task in meta domain adaptation should be seen as a pair of source task and target task.
The problem with the current model is that during training, it is trained to target at one specific type of test domain--the generator network G aims to generated images that align with the unsupervised  test domain X_test. Thus, the trained model will also only be able to handle one test domain, not much different than regular meta-learning models.
In short, the meta-learning part stays in the regular few-shot learning module (which is implemented as a prototypical network), and has nothing related to domain adaptation. Therefore, the paper cannot be qualified for ``meta domain adaptation’’ and has very limited novelty in terms of its contribution to meta-learning; however, the combination of domain adaptation and few-shot learning is fair. For the rest of my review, I will treat the paper as “few-shot learning with domain adaptation” for more appropriate analysis.
For the experiments, there seems to have a great win of the proposed algorithm against the baselines. However, I think since this is few-shot learning with domain adaptation, there is no domain adaptation baselines being mentioned in comparison. Specifically, what if the few-shot learning component is removed, and the network is trained with standard domain adaptation. Then use the same network to extract the features and then using the nearest neighbor to retrieve the classes. Also it seems that the regular batch normalization could be very sensitive to domain shifts, and it would be good if the authors can test other normalization schemes such as layer/group normalization as baselines.
Another concern is that the evaluation of domain adaptation does not have much varieties. Only two domains shifts are evaluated in the paper, specifically Omniglot + BSD500 and Office-Home. BSD 500 only contains 500 images, and it would be good if more diverse set of images are considered. Other domain transfer settings such as synthetic rendered vs. real (e.g. visDA challenge) could have been considered.
In conclusion, the paper presents a interesting combination of ProtoNet + Adversarial DA + Cycle consistency. However, unlike as advertised, the paper does not address the domain shift issue in meta-learning, and the experiments lack thorough evaluation as the paper considers itself as a meta-learning paper and only compares to other meta-learning approaches without much comparison to domain adaptation papers. Therefore, I recommend reject.
---
Note: after reading the comments updated by authors, I remain my opinions: even though exact meta-testing data is unseen during training, the domain is seen during training, and therefore it cannot be qualified for being "meta domain adaptation".
===
After rebuttal:
I would like to thank the authors for the response and updating the draft. They have addressed 1) the title issue and 2) adding domain adaptation baselines. Considering these improvements, I would like to raise the score to 5, since the setting of combining few-shot learning and domain adaptation is interesting and the proposed model outperforms the baselines.
However, my criticisms remain that the paper is a simple combination of cycle GAN and prototypical networks, and lacks new insights/novelty. The experiments use fairly small datasets, where the performance can be largely influenced by how good the feature extractor backbone is (e.g. training on more data and using deeper architecture would warrant better performance, and thus may change the conclusion).


Automatic search of neural network architectures is a standing research topic. In addition to the fact that it presents a faster alternative to hand-designed architectures, it can improve their efficiency and for instance generate Convolutional Neural Networks (CNN) adapted for mobile devices. In this paper, we present a multi-objective neural architecture search method to find a family of CNN models with the best accuracy and computational resources tradeoffs, in a search space inspired by the state-of-the-art findings in neural search. Our work, called Dvolver, evolves a population of architectures and iteratively improves an approximation of the optimal Pareto front. Applying Dvolver on the model accuracy and on the number of floating points operations as objective functions, we are able to find, in only 2.5 days 1 , a set of competitive mobile models on ImageNet. Amongst these models one architecture has the same Top-1 accuracy on ImageNet as NASNet-A mobile with 8% less floating point operations and another one has a Top-1 accuracy of 75.28% on ImageNet exceeding by 0.28% the best MobileNetV2 model for the same computational resources.

The paper proposes a multi-objective search algorithm that designs resource-efficient convolutional architectures. The key idea is to maintain a population of networks and to iteratively approach the Pareto front through evolution. The normal & reduction cells are searched on CIFAR-10 and then transferred to ImageNet. The resulting architectures empirically lead to better trade-offs than other baselines.
Pros:
The paper is well-written and easy to comprehend.
Results are competitive against strong baselines such as NASNet.
Resource budgets are handled in a principled manner with multi-objective optimization.
Cons:
My main concerns are on the technical novelty and experimental comparison.
Technical novelty:
The proposed algorithm seems highly similar to the existing multi-objective NAS algorithms, especially the ones based on Pareto optimality [1,2,3]. In Sect 2, the authors state that the main difference from prior works such as [2] and [3] is the usage of a different and larger search space and large-scale experiments. However, both aspects are of limited technical novelty.
Experimental comparison:
In sect 3.3, the authors say “we noticed that the original NASNet search space can greatly benefit from extra connections from any given block”. If the proposed algorithm was investigated in an enhanced version of the NASNet space, it would be unclear whether we should attribute the reported performance to the proposed multi-objective evolution or this additional search space engineering. It would be better to report the results using the original space as well for fair comparison.
The main claimed contribution is a multi-objective evolutionary algorithm. To demonstrate its effectiveness, it would be necessary to compare against existent multi-objective NAS strategies in the literature. Most of those strategies (e.g., scalarization, weighted product method) should be straightforward to implement on top of the current search space. The current results are less convincing since the authors only compared their method against single-objective baselines (e.g. NASNet, PNAS, AmoebaNet) which are completely unaware of additional dimensions of the desired objectives.
The networks are searched on CIFAR-10 and then transferred to ImageNet. Unlike most prior works (including the ones focusing on resource-constrained NAS), the authors did not the final performance of their architecture on CIFAR-10. It would be informative to report the CIFAR-10 results as well.
Other suggestions & questions:
The authors did not report their training setup for ImageNet. It would be good to include those details to ensure the readers are informed should there are any additional augmentations.
“uniform mutation and a crossover probability of 0.1” (sect 4.1)
It would be better to included more details on these evolution forces for reproducibility. These are also important component of the proposed algorithm.
“We manually select 3 architectures that we will be fully train on ImageNet in Section 4.2” (sect 4.1)
I believe this part needs more clarifications since there can be a large number of architectures on the Pareto front. What’s the criteria for manual selection?
[1] Elsken, Thomas, Jan Hendrik Metzen, and Frank Hutter. "Multi-objective architecture search for cnns." arXiv preprint arXiv:1804.09081 (2018).
[2] Kim Ye-Hoon, Reddy Bhargava, Yun Sojung, and Seo Chanwon. NEMO: Neuro-Evolution with Multiobjective Optimization of Deep Neural Network for Speed and Accuracy. ICML’17 AutoML Workshop, 2017.
[3] Dong, Jin-Dong, et al. "DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures." arXiv preprint arXiv:1806.08198 (2018).


Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.

Pros:
This paper uses kernel mappings between any two layers for weight initialisation. Using the representer theorem, a proper distribution for weights is constructed in H_{k_i} instead of being learned by \phi_i, and then is formulated as a GP.
Cons:
However, there are some key issues.
1. The so-called “infinite width” is just yielded by kernels in RKHS for weight initialization. For practical implementation, the authors use this scheme with random Fourier features to construct finite-width network. A key issue is that how to guarantee that the approximated weights are still in the same space? For example, weights can be in RKHS, but their approximation might be not in RKHS. See in [S1] for details.

[S1] Generalization Properties of Learning with Random Features, NIPS 2017.

2. Experimental part is not very convincing. First, the authors just compare different initialization schemes. The used architectures are simple and not representative. Second, the overall performance is not satisfactory, and the compared classification datasets are quite small. Overall, the experimental results are inadequate and unconvincing.
Summary:
The paper attempts to proposal a weight initialization scheme to enable infinite deep infinite-width networks. However, there are some key issues not address such as whether the approximated weights are still in the same space and the limited experimental results.
Response to rebuttal:
The authors have addressed my question about the weights being still in the same RKHS. I still think the motivation and experiments are not very satisfactory.
Therefore the paper is very borderline. However, I would like to bump my rating a bit higher.


This paper proposes a new method to drastically speed up deep reinforcement learning (deep RL) training for problems that have the property of state-action permissibility (SAP). Two types of permissibility are defined under SAP. The first type says that after an action a t is performed in a state s t and the agent reaches the new state s t+1 , the agent can decide whether the action a t is permissible or not permissible in state s t . The second type says that even without performing the action a t in state s t , the agent can already decide whether a t is permissible or not in s t . An action is not permissible in a state if the action can never lead to an optimal solution and thus should not be tried. We incorporate the proposed SAP property into two state-of-the-art deep RL algorithms to guide their state-action exploration. Results show that the SAP guidance can markedly speed up training.

This paper proposed the concept of state-action permissibility (SAP). Given a user-defined type 1 SAP function, the algorithm learns a classifier to predict whether an action at a given state is permissible or not. Based on this prediction, the reinforcement learning (RL) algorithms can limit the exploration only to the permissible actions, and thus greatly reduce the cost of learning. The proposed algorithms are tested on two simple tasks, both of which have the same flavor of following a predefined track.
Although the results of the experiments show that SAP helps to speed up RL, I think that the application of SAC is very narrow. It is extremely difficult to define an AP1 function in general. For example, for most of the OpenAI gym environments (such as half-cheetah, ants or humanoid), it is not clear to me how to manually define an AP1 function. It would be more convincing if the paper can apply the proposed techniques to some of the benchmark OpenAI gym environments.
Even for the lane following task described in the paper, the AP1 function in eq. 5 is limited and eliminates many good solutions. It constrains that the action should not lead to more deviations to the center line in the next time step. This greedy constraint will not work in more interesting driving scenarios. For example in a sharp turn, if the curvature of the lane is too large for the car to follow, a common strategy (that can be learned by vanilla RL algorithms) is to first drive to the outer side of the lane before the turn, cut to the inner side at the turn and exit the turn to the outer side. This optimal solution to negotiate a tight turn is completely eliminated by the user-defined AP1 function (eq. 5).
The idea of AP1 is somewhat contradictory to the philosophy of reinforcement learning. AP1 is a greedy decision based on the next step while RL optimizes for the accumulated reward over many steps. RL allows taking an action that will sacrifice the immediate reward (e.g. deviate from the center line of a lane) in the next step but can accumulated more reward in the long run (successfully drive along a tight turn). In most of cases, by looking at the next state, it is just not possible to predict whether a specific action cannot lead to the optimal long-term reward (SAP).
For the above reasons, I think that the application of SAP would be very narrow, especially for reinforcement learning. I would not recommend accepting this paper at this time.


A new statistical procedure (Model-X Candès et al. (2018)) has provided a way to identify important factors using any supervised learning method controlling for FDR. This line of research has shown great potential to expand the horizon of machine learning methods beyond the task of prediction, to serve the broader needs in scientific researches for interpretable findings. However, the lack of a practical and flexible method to generate knockoffs remains the major obstacle for wide application of Model-X procedure. This paper fills in the gap by proposing a model-free knockoff generator which approximates the correlation structure between features through latent variable representation. We demonstrate our proposed method can achieve FDR control and better power than two existing methods in various simulated settings and a real data example for finding mutations associated with drug resistance in HIV-1 patients.

This manuscript tackles an important problem, namely, generating the knockoff procedure for FDR-controlled feature selection so that it can work on any data, rather than only for data generated from a Gaussian (as in the original work) or several specific other cases (mixtures of Gaussians or HMMs).  The basic idea is to exploit ideas from variational autoencoders to create a generic knockoff generation mechanism. Specifically, the main idea of the paper is to map the input covariates X into latent variable Z using a variational auto-encoder, generate the knockoffs \tilde{Z} in the latent space, and then map \tilde{Z} back to the input space to get the knockoffs \tilde{X}. The authors claim that their contribution in threefold:
(1) Given the assumption is valid that X is mutually independent conditional on Z, the generated knockoffs \tilde{X} is proved to be valid in terms of satisfying the necessary swap condition.
(2) Given (1) holds, and given that the discrepancy (measured by KL-divergence) between the true conditional probability Q(Z|X) and its estimate using variational auto-encoder is bounded by o((nlogp)^{-1/2}), the FDR is also bounded.
(3) The proposed knockoffs generating procedure can achieve controlled FDR and better power.
In agreement with the above intuition, I have major concerns about the correctness of the paper.
The cornerstone of the proof in contribution (1) relies on the assumption that X is mutually independent conditional on Z. However, this assumption is invalid if there are dependencies between x_i and x_j. Therefore, the proposed knockoffs \tilde{X} cannot be proved valid by Theorem 2.1.
The erroneous proof in contribution (1) leads to the failure of contribution (2) stated in Theorem 2.2. The FDR is no longer controlled. Intuitively, according to algorithm 1, \tilde{Z} and \tilde{X} are expected to have the same distribution as Z and X, respectively; therefore, the FDR cannot be controlled.
The experimental results are suspicious. In general, it seems fishy that the proposed VAE approach outperforms Model X in the fully Gaussian case.  In this setting, Model X should have an advantage, since it is specifically designed for Gaussian generated data.  Related to this, the text is confusing: "Since the data were not Gaussian, the second-order matching method has the lowest power. The assumptions of the Fixed knockoff generations holds for the Gaussian cases, …" In the figure, the second-order matching method has low power even in the Gaussian case.
Minor comments:
p. 2: The exposition should explain the motivation for Knockoff+.
The manuscript contains numerous grammatical errors, a few examples of which are below:
p. 1: "biological linked" -> "biologically linked"
p. 1: "associated certain" -> "associated with a certain"
p. 1: "showed light" -> "shed light"
p. 1: "which has very limited" -> "which has limited"
p. 1: "leveraging on the power of of" -> "leveraging the power of"


PROBABILISTIC PLANNING WITH SEQUENTIAL MONTE CARLO METHODS
In this work, we propose a novel formulation of planning which views it as a probabilistic inference problem over future optimal trajectories. This enables us to use sampling methods, and thus, tackle planning in continuous domains using a fixed computational budget. We design a new algorithm, Sequential Monte Carlo Planning, by leveraging classical methods in Sequential Monte Carlo and Bayesian smoothing in the context of control as inference. Furthermore, we show that Sequential Monte Carlo Planning can capture multimodal policies and can quickly learn continuous control tasks.

The authors formulate planning as sampling from an intractable distribution motivated by control-as-inference, propose to approximately sample from the distribution using a learned model of the environment and SMC, then evaluate their approach on 3 Mujoco tasks. They claim that their method compares favorably to model-free SAC and to CEM and random shooting (RS) planning with model-based RL.
This is an interesting idea and an important problem, but there appear to be several inconsistencies in the proposed algorithm and the experimental results do not provide compelling support for the algorithm. In particular,
Levine 2018 explains that with stochastic transitions, computing the posterior leads to overly optimistic behavior because the transition dynamics are not enforced, whereas the variational bound explicitly enforces that. Is that an issue here?
The value function estimated in SAC is V^\pi the value function of the current policy. The value function needed in Sec 3.2 is a different value function. Can the authors clarify on this discrepancy?
The SMC procedure in Alg 1 appears to be incorrect. It multiplies the weights by exp(V_{t+1}) before resampling. This needs to be accounted for by setting the weights to exp(-V_{t+1}) instead of uniform. See for example auxiliary particle filters.
The experimental section could be significantly improved by addressing the following points:
* How was the planning horizon h chosen? Is the method sensitive to this choice? What is the model accuracy?
* Does CEM use a value function? If not, it seems like a reasonable baseline to consider CEM w/ a value function to summarize the values beyond the planning horizon. This will evaluate whether SMC or including the value function is important.
* Comparing to state-of-the-art model-based RL (e.g., one of Chua et al. 2018, Kurutach et al. 2018, Buckman et al. 2018).
* How were the task # of steps chosen? They seem arbitrary. What is the performance at 1million and 5million steps?
* Was SAC retuned for this small number of samples/steps?
* Clarify where the error bars come from in Fig 5.2 in the caption.
At the moment, SMCP is within the error bars of a baseline method.
Comments:
In the abstract, the authors claim that the major challenges in planning are: 1) model compounding errors in roll-outs and 2) the exponential search space. Their method only attempts to address 2), is that correct? If so, can the authors state that explicitly.
Recent papers (Chua et al. 2018, Kurutach et al. 2018, Buckman et al. 2018, Ha and Schmidhuber 2018) all show promising model-based results on continuous state/action tasks. These should be mentioned in the intro.
The connection between Gu et al.'s work on SMC and SAC was unclear in the intro, can the authors clarify?
For consistency, ensure that sums go to T instead of \infty.
I found the discussion of SAC at the end of Sec 2.1 confusing. As I understand SAC, it does try to approximate the gradient of the variational bound directly. Can the authors clarify what they mean?
At the end of Sec 2.2, the authors claim that the tackle the particle degeneracy issue (a potentially serious issue) by "selecting the temperature of the resampling distribution to not be too low." I could not find further discussion of this anywhere in the paper or appendix.
Sec 3.2, mentions an action prior for the first time. Where does this come from?
Sec 3.3 derives updates assuming a perfect model, but we learn a model. What are the implications of this?
Please ensure the line #'s and the algorithm line #'s match.
Model learning is not described in the main text though it is a key component of the algorithm. The appendix lacks details (e.g., what is the distribution used to model the next state?) and contradicts itself (e.g., one place says 3 layers and another says 2 layers).
In Sec 4.1, a major difference between MCTS and SMC is that MCTS runs serially, whereas SMC runs in parallel. This should be noted and then it's unclear whether SMC-Planning should really be thought of as the maximum entropy tree search equivalent of MCTS.
In Sec 4.1, the authors claim that Alpha-Go and SMCP learn proposals in similar ways. However, SMCP minimizes the KL in the reverse direction (from stated in the text). This is an important distinction.
In Sec 4.3, the authors note that Gu et al. learn the proposal with the reverse KL from SMCP. VSMC (Le et al. 2018, Naesseth et al. 2017, Maddison et al. 2017) is the analogous work to Gu et al. that learn the proposal using the same KL direction as SMCP. The authors should consider citing this work as it directly relates to their algorithm.
In Sec 4.3, the authors claim that their direction of minimizing KL is more appropriate for exploration. Gu et al. suggest the opposite in their work. Can the author's justify their claim?
In Sec 5.1, the authors provide an example of SMCP learning a multimodal policy. This is interesting, but can the authors explain when this will be helpful?
====
11/26
At this time, the authors have not responded to reviews. I have read the other reviews. Given the outstanding issues, I do not recommend acceptance.
12/7
After reading the author's response, I have increased my score. However, baselines that establish the claim that SMC improves planning which leads to improved control are missing (such as CEM + value function). Also, targeting the posterior introduces an optimism bias that is not dealt with or discussed.


AD-VAT: AN ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING
Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations. Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios. However, during training, such method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker's generalization on unseen object moving patterns. To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT. In AD-VAT, both the tracker and the target are approximated by end-to-end neural networks, and are trained via RL in a dueling/competitive manner: i.e., the tracker intends to lockup the target, while the target tries to escape from the tracker. They are asymmetric in that the target is aware of the tracker, but not vice versa. Specifically, besides its own observation, the target is fed with the tracker's observation and action, and learns to predict the tracker's reward as an auxiliary task. We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker. To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target. The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios. For supplementary videos, see: https://www.youtube. com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS

This paper presents a simple multi-agent Deep RL task where a moving tracker tries to follow a moving target. The tracker receives, from its own perspective, partially observed visual information o_t^{alpha} about the target (e.g., an image that may show the target) and the target receives both observations from its own perspective o_t^{beta} and a copy of the information from the tracker's perspective. Both agents are standard convnet + LSTM neural architectures trained using A3C and are evaluated in 2D and 3D environments. The reward function is not completely zero-sum, as the tracked agent's reward vanishes when it gets too far from a reference point in the maze.
The work is very incremental over Luo et al (2018) "End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning", as the only two additions are extra observations o_t^{alpha} for the target, and a reward function that has a fudge factor when the target gets too far away. Citing Sun Tzu's "Art of War" (please use the correct citation format) is not convincing enough for adding the tracker's observations as inputs for the target agent. Should not the asymmetrical relationship work the other way round, with the tracker knowing more about the target?
Experiments are conducted using two baselines for the target agent, one a random walk and another an agent that navigates to a target according to a shortest path planning algorithm. The ablation study shows that the tracker-aware observations and a target's reward structure that penalizes when it gets too far do help the tracker's performance, and that training the target agent helps the tracker agent achieve higher scores. The improvement is however quite small and the task is ad-hoc.

The paper would have benefitted from a proper analysis of the trajectories taken by the adversarial target as opposed to the heuristic ones, and from comparison with non-RL state-of-the-art on tracking tasks. Further multi-agent tasks could also have been considered, such as capture the flag tasks as in "Human-level performance in first-person multiplayer games with population-based deep reinforcement learning".


PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND PRIORS
We study the problem of generating adversarial examples in a black-box setting in which only loss-oracle access to a model is available. We introduce a framework that conceptually unifies much of the existing work on black-box attacks, and we demonstrate that the current state-of-the-art methods are optimal in a natural sense. Despite this optimality, we show how to improve black-box attacks by bringing a new element into the problem: gradient priors. We give a bandit optimization-based algorithm that allows us to seamlessly integrate any such priors, and we explicitly identify and incorporate two examples. The resulting methods use two to four times fewer queries and fail two to five times less than the current state-of-the-art.

This paper formulates the black-box adversarial attack as a gradient estimation
problem, and provide some theoretical analysis to show the optimality of an
existing gradient estimation method (Neural Evolution Strategies) for black-box
attacks.
This paper also proposes two additional methods to reduce the number of queries
in black-box attack, by exploiting the spacial and temporal correlations in
gradients. They consider these techniques as priors to gradients, and a bandit
optimization based method is proposed to update these priors.
The ideas used in this paper are not entirely new. For example, the main
gradient estimation method is the same as NES (Ilyas et al. 2017);
data-dependent priors using spatially local similarities was used in Chen et
al. 2017.  But I have no concern with this and the nice thing of this paper is
to put these tricks under an unified theoretical framework, which I really
appreciate.
Experiments on black-box attacks to Inception-v3 model show that the proposed
bandit based attack can significantly reduces the number of queries (2-4 times
fewer) when compared with NES.
Overall, the paper is well written and ideas are well presented.
I have a few concerns:
1) In Figure 2, the authors show that there are strong correlations between the
gradients of current and previous steps. Such correlation heavily depends on
the selection of step size.  Imagine that the step size is sufficiently large,
such that when we arrive at a new point for the next iteration, the
optimization landscape is sufficiently changed and the new gradient is vastly
different than the previous one. On the other hand, when using a very small
step-size close to 0, gradients between consecutive steps will be almost the
same. By changing step-size I can show any degree of correlation.  I am not
sure if the improvement of Bandit_T comes from a specific selection of
step-size. More empirical evidence on this need to be shown - for example, run
Bandit_T and NES with different step sizes and observe the number of queries
required.
2) This paper did not compare with many other recent works which claim to
reduce query numbers significantly in black-box attack. For example, [1]
proposes "random feature grouping" and use PCA for reducing queries, and [2]
uses a good gradient estimator with autoencoder. I believe the proposed method
can beat them, but the authors should include at least one more baseline to
convince the readers that the proposed method is indeed a state-of-the-art.
3) Additionally, the results in this paper are only shown on a single model
(Inception-v3), and it is hard to compare the results directly with many other
recent works. I suggest adding at least two more models for comparison (most
black-box attack papers also include MNIST and CIFAR, which should be easy to
add quickly). These numbers can be put in appendix.
Overall, this is a great paper, offering good insights on black-box adversarial
attack and provide some interesting theoretical analysis. However currently it
is still missing some important experimental results as mentioned above, and
not ready to be published as a high quality conference paper. I conditionally
accept this paper as long as sufficient experiments can be added during the
discussion period.
[1] Exploring the Space of Black-box Attacks on Deep Neural Networks, by Arjun
Nitin Bhagoji, Warren He, Bo Li and Dawn Song, https://arxiv.org/abs/1712.09491
(conference version accepted by ECCV 2018)
[2] AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking
Black-box Neural Networks, by Chun-Chen Tu, Paishun Ting, Pin-Yu Chen, Sijia
Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, Shin-Ming Cheng,
https://arxiv.org/abs/1805.11770
==========================================
After discussing with the authors, they provided better evidence to support the conclusions in this paper, and fixed bugs in experiments. The paper looks much better than before. Thus I increased my rating.


Deep convolutional network architectures are often assumed to guarantee generalization for small image translations and deformations. In this paper we show that modern CNNs (VGG16, ResNet50, and InceptionResNetV2) can drastically change their output when an image is translated in the image plane by a few pixels, and that this failure of generalization also happens with other realistic small image transformations. Furthermore, we see these failures to generalize more frequently in more modern networks. We show that these failures are related to the fact that the architecture of modern CNNs ignores the classical sampling theorem so that generalization is not guaranteed. We also show that biases in the statistics of commonly used image datasets makes it unlikely that CNNs will learn to be invariant to these transformations. Taken together our results suggest that the performance of CNNs in object recognition falls far short of the generalization capabilities of humans.

Paper summary:
As is made clear in the title, this paper sets out to answer the following question: “Why do deep convolutional networks generalize so poorly to small image transformations?”. It focuses on natural image transformations on translation and scaling (rotation is missing though).
The paper proposes two main explanations:
-	Strided convolution, called subsampling in the paper, ignores the classical sampling theorem,
-	CNNs will not learn invariance because of the (photographers') biases contained in the datasets.
On a general level, the paper is a good read, it is well written and the figures clearly convey the message they’re intended to. Adversarial attacks and robustness of CNNs in general is a very interesting and important topic in ML. The originality of this work is in the approach of the problem, the paper tries to explain the reasons why CNNs are vulnerable. Related works put more emphasis on coming up with novel attacks/defense strategies. Considering natural attacks as done in this submission is particularly interesting as it is probably a more surprising shortcoming of CNNs compared to optimally designed attacks or highly unnatural perturbations. The argument about subsampling (stride) being the reason of not having translational invariance is nice, especially the theoretical insight with the Shannon-Nyquist theorem and the more figurative example on part detectors. There are nevertheless a few major concerns about this work:
Major Concerns:
Theoretical arguments:
The theoretical argument made in this paper is interesting but to make the point stronger a more in-depth explanation would be needed.
-	The step from Eq (2) to Eq (3) is not entirely clear “K does not depend on x_i”, maybe one extra sentence to explain this step would be useful.
-	Terms introduced such as the basis function B and the set of transformations T could be better defined.
-	For the extension to other types of transformations “While the claim focuses on global translation, it can also be extended to piecewise constant transformations.” it would be important to point out what type of natural transformations can be included in this set.
Empirical evidence:
Experiments are not fully convincing. Additional empirical evidence would be beneficial and necessary to support the claims of this:
-	“A natural criticism of these results is that they are somehow related to the image resizing and inpainting procedures that we used.” This is a very good point and the authors following arguments are not fully convincing. Results with different transformation procedures mentioned in the rest of the paragraph (and probably more) should be included to convince the reader.
-	The theoretical argument that translation invariance is not guaranteed because of the stride (subsampling) is not fully convincing and needs further explanation and experimental verification. In fact, feature maps of the CNNs that the authors consider do indeed contain many high frequencies.
-	The argument made in part 4 about the photographer’s bias seems valid for general natural transformations, but it does not apply to small transformations such as 1-pixel translations presented in the paper. Also, evidence that datasets without (or less) photographers' bias are less susceptible to natural attacks would make the argument in the paper a lot stronger.
-	When using 6x6 avg pooling for the VGG16 architecture ”recognition performance decreases somewhat” . Results are only preliminary in the paper, but this statement needs a more thorough experimental backing. It should come with convincing quantitative evidence.
-	Please include some results or citation on other work about test time augmentation to support the statement “still only provides partial invariance”.
References and phrasing:
Generally previous work is well referenced in this paper, although there are some formulations that can be slightly modified to make a clear distinction between what is novel and what is previous work:
-	As is very well shown in the introduction, there is a lot of work on generating adversarial examples that drastically change the output of a CNN. This should be made clear in the abstract, in fact the sentence “In this paper we show that modern CNNs [...] also happens with other realistic small image transformations”  seems to indicate that this is the novel work in the paper. This is also why I believe the first sentence “Deep convolutional network architectures are often assumed to guarantee generalization for small image translations and deformations.” is somewhat contestable.
-	“We find that modern deep CNNs are not invariant to translations, scalings and other realistic image transformations” as the paper points out earlier this is not a novel finding, so I would use a formulation that makes that clear and gives more emphasis to your own arguments as of why this happens.
Further Comments :
-	Part 5 "Implications for Practical Systems" could be moved to discussion as there is no new point and it seems more a reflection on what was already stated.
-	The final sentence of the abstract “Taken together our results suggest that the performance of CNNs in object recognition falls far short of the generalization capabilities of humans.” is not necessary, this is clearly true but it isn’t really contested in the ML community.
-	“despite the architecture being explicitly designed to provide such invariances” I agree that this has motivated the use and design of CNNs in the first place, but modern architectures are mostly designed to surpass the results on the common benchmarks rather than to provide such invariances.
-	”jaggedness is greater for the modern, deeper, networks compared to the less modern VGG16 network” might be worth interesting to consider if the residuals have anything to do with it.


We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons. Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information. A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones. Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns. In this manner, we enable the slope of the activation function to be context dependent. This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.

Summary: this submission proposes a modification of neural network architectures that allows the modulation of activation functions of a given layer as a function of the activations in the previous layer. The author provide different version of their approach adapted to CNN, DenseNets and LSTM, and show it outperforms a vanilla version of these algorithms.
Evaluation: In the classical context of supervised learning tasks investigated in this submission, it is unclear to me what could be the benefit of introducing such “modulators”, as vanilla ANNs already have the capability of modulating the excitability of their neurons. Although the results show significant, but quite limited, improvements with respect to the chosen baseline, more extensive baseline comparisons are needed.
Details comments:
1.	Underlying principles of the approach
It is unclear to me why the proposed approach should bring a significant improvement to the existing architectures. First, from a neuroscientific perspective, neuromodulators allow the brain to go through different states, including arousal, sleep, and different levels of stress. While it is relatively clear that state modulation has some benefits to a living system, it is less so for an ANN focused on a supervised learning task. Why should the state change instead of focusing on the optimal way to perform the task? If the authors want to use a neuroscientific argument, I would suggest to elaborate based on the precise context of the tasks they propose to solve.
In addition, as mentioned several times in the paper, neuromodulation is frequently associated to changes in cell excitability. While excitability is a concept that can be associated to multiple mechanisms, a simple way to model changes in excitability is to modify the threshold that must be reached by the membrane potential of a given neuron in order for the cell to fire. Such simple change in excitability can be easily implemented in ANNs architectures by affecting one afferent neuron in the previous layer to the modification of this firing threshold (simply adding a bias term). As a consequence, if there is any benefit to the proposed architecture, it is very likely to originate specifically from the multiplicative interactions used to implement modulation in this paper. However, approximation of such multiplicative interactions can also be implemented using multiple layers network equipped with non-linear activations. Overall, it would be good to discuss these aspects in great detail in the introduction and/or discussion of the paper, and possibly find a more convincing justification for the approach.
2.	Weak baseline comparison results
In the CNN experiments, modulated networks are only compared with a single vanilla counterpart equipped with ReLu. There are at least two obvious additional baseline comparison that would be useful: what if the Re-Lu activations are replaced with fixed sigmoids? And what if batch-normalization is switched on/off (I could not find whether it was used at all). Indeed it, we should exclude benefits that are simply due to the non-linearity of the sigmoid, and batch normalization also implements a form of modulation at training that may provide benefits equivalent to modulation (or on the contrary, batch norm could implement a modulation in the wrong way). It would be better to look at all possible combinations of these architecture choices.
Due to lack of details in the paper and my personal lack of expertise in LSTMs, I will not comment on baselines for that part but I assume similar modifications can be done.
Overall, given the weak improvements in performance, it is questionable whether this extra degree of complexity should be added to the architecture. Additionally, I could not find the precise description of the statistical tests performed. Ideally, the test, the number of samples, the exact p-value, and whether the method of correction for multiple comparison should be included each time a p-value is mentioned.


We present a soft, spatial, sequential, top-down attention model (S3TA). This model uses a soft attention mechanism to bottleneck its view of the input. A recurrent core is used to generate query vectors, which actively select information from the input by correlating the query with input-and space-dependent key maps at different spatial locations. We demonstrate the power and interpretabilty of this model under two settings. First, we build an agent which uses this attention model in RL environments and show that we can achieve performance competitive with state-of-the-art models while producing attention maps that elucidate some of the strategies used to solve the task. Second, we use this model in supervised learning tasks and show that it also achieves competitive performance and provides interpretable attention maps that show some of the underlying logic in the model's decision making.

[Summary]
This paper proposed a soft, spatial, sequential, top-down attention model which enable the agent and classifier actively select important, task relative information to generate the appropriate output. Given the observations, the proposed method uses a convLSTM to produce the key and value tensor. Different from multi-head attention, the query vector is generated in a top-down fashion. The authors proposed to augment the spatial feature with Fourier bases which is similar to previous work. The authors verify the proposed method on both reinforcement learning and supervised learning. On reinforcement learning, the proposed method outperformed the feedforward baseline and LSTM baseline. On reinforcement learning task, the proposed method achieves compelling result with more interpretable attention map that shows the model's decision.
[Strength]
1: The proposed model is a straightforward extension of the multi-head attention to visual input. Compare to multi-head attention, it generates the query vector in a top-down manner instead of pure bottom up, and the authors verify the proposed choice is better than LSTM baseline empirically.
2: The authors verify the proposed method by extensive experiments on reinforcement learning tasks and also try supervised learning tasks. The attention visualization and human normalized scores for experts on ATARI show the effectiveness of the proposed method.
[Weakness]
1: The soft spatial top-down attention is very common in vision and language domain, such as VQA. As the authors mentioned, the proposed method is very similar with MAC for CLEVER. The sequential attention is also explored in previous VQA work. Thus the novelty of the proposed method is limited.
2: Multi-head attention for NLP tasks are usually composed with multiple layers. Will more layer of attention help the performance? The paper is less of this ablation study.
3: The proposed method is worse compared with other baselines on supervised learning tasks, on both imagenet classification and kinetics. I wonder whether the recurrent process is required for those tasks? On table 2, we can observe that with sequence length 8, the performance is much worse,  this may be caused by overfitting.
4: If the recurrent attention is more interpretable, given other visualization methods, such as gradcam, I wonder what is advantage?
5: I would expect that the performance on Kinetics dataset is better since sequential attention is required on video dataset. However, the performance is much worse compare of the baseline in the dataset. I wonder what is the reason? is there ablation study or any other results on this dataset?


ALGORITHMIC FRAMEWORK FOR MODEL-BASED DEEP REINFORCEMENT LEARNING WITH THEORETI- CAL GUARANTEES
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves stateof-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.

The paper proposed a framework to design model-based RL algorithms. The framework is based on OFU and within this framework the authors develop an algorithm (a variant of SLBO) achieving SOTA performance on MuJoCo tasks.
The paper is very well written and the topic is important for the RL community. The authors do a good job at covering related works, the bounds are very interesting and the results quite convincing.
Questions/comments to the authors:
1) In footnote 3 you state that "[...] we only need to approximate the dynamical model accurately on the trajectories of the optimal policy". Why only of the optimal policy? Don't you also need an accurate dynamic model for the current policy to perform a good policy improvement step?
2) A major challenge in RL is that the state distribution \rho^\pi changes with \pi and it is usually very hard to estimate. Therefore, many algorithms assume it does not change if the policy is subject to small changes (examples are PPO and TRPO). In Eq 4.3 it seems that you also do something similar, fixing \rho^\pi and constraining the KL of \pi (and not of the joint distribution p(s,a)). Am I correct? Can you elaborate it a bit more, building a connection with other RL methods?
3) In Eq. 6.1 and 6.2 you minimize the H-loss, defined as the prediction error of your model. Recently, Pathak et al. used the same loss function in many papers (such as Curiosity-driven Exploration by Self-supervised Prediction) and your Eq. 6.2 looks like theirs. The practical implementation of your algorithm looks very similar to theirs too. Can you comment on that?
4) If I understood it correctly, your V-function directly depends on your model, i.e., you have V(M(s)) and you learn the model M parameters to maximize V. This means that you want to learn the model that, together with the policy, maximizes V. Am I correct? Can you comment a bit more on that? Did you try to optimize them (V and M) separately, i.e., to add a third parameter to learn (the V-function parameters)?
5) How does you algorithm deal with environmental noise? The tasks used for the evaluation are all deterministic and I believe that this heavily simplifies the model learning. It would be interesting an evaluation on a simple problem (for example the swing-up pendulum) in the presence of noise on the observations and/or the transition function.
6) I appreciate that you provide many details about the implementation in the appendix. Can you comment a bit more? Which are the most important hyperparameters? The number of policy optimization n_policy or of model optimization n_model? You mention that you observed policy overfitting at the first iterations. Did you also experience model overfitting? Did normalizing the state help a lot?


In machine learning tasks, overtting frequently crops up when the number of samples of target domain is insufficient, for the generalization ability of the classifier is poor in this circumstance. To solve this problem, transfer learning utilizes the knowledge of similar domains to improve the robustness of the learner. The main idea of existing transfer learning algorithms is to reduce the dierence between domains by sample selection or domain adaptation. However, no matter what transfer learning algorithm we use, the difference always exists and the hybrid training of source and target data leads to reducing fitting capability of the learner on target domain. Moreover, when the relatedness between domains is too low, negative transfer is more likely to occur. To tackle the problem, we proposed a two-phase transfer learning architecture based on ensemble learning, which uses the existing transfer learning algorithms to train the weak learners in the first stage, and uses the predictions of target data to train the final learner in the second stage. Under this architecture, the fitting capability and generalization capability can be guaranteed at the same time. We evaluated the proposed method on public datasets, which demonstrates the effectiveness and robustness of our proposed method.

This paper proposed to solve the instance-based transfer learning and feature-based transfer learning by stacking with a two-phase training strategy. The source data and target data are hybrid together first to train weak learners, and then the ensembled super learner is utilized to get the final prediction. Details for the stacking process are provided. Experimental results on MNIST-USPS, COIL, and Office-Caltech datasets show the proposed method can boost the performance, compared to TrAdaboost.
Pros:
The paper proposes to using stacking or ensembling to solve the domain adaptation problem, which shows some insight for further domain adaptation research.
Cons:
1. One of the main issues of this paper is the lack of novelty. The framework is incremented from the previous domain adaptation method such as TrAdaboost or BDA. For feature-based transfer learning, Equation (7)(8)(9) directly from the previous method.
2. Some arguments in this paper are not solid. For example, in the abstract,   the authors claim that under the two-stage training architecture, the fitting capability and generalization capability can be guaranteed at the same time. However, this is not well-justified in the following literature. Another example is "the settings of \lamda and N should be taken into consideration, if \lambda is too large, the performance of each learner can't be guaranteed, if \lambda is too small, training data can't be diversified enough" (page 7line 9~11)
3. This paper is weakened by the experimental part. Firstly, only TraDaboost method is used as a baseline. The paper can be largely improved by comparing with the state-of-the-art ensembling method for domain adaptation, for example:
Self-ensembling for visual domain adaptation, Geoff French,  ICLR 2018.
Secondly, the datasets used in this paper is small-scale and biased. It would be exciting to see how the proposed method will perform on the state-of-the-art large-scale domain adaptation dataset, for example, Office-Home dataset, Syn2Real dataset.
 Others:
1. Some terminologies used in this paper are confusing: (1) the h_t and c are not defined in Equation (2). in Algorithm 2, how to construct kernel matrix K_t using k_t?
2. The written of this paper can be largely improved. Some sentences are grammarly mistaken. Typos examples:
Abstract line 1: overtting -> overfitting
Section 2.1, we use TraAdaboost -> We use TrAdaboost
3. The citation style used in this paper is not correct.
Problems:
1. In section 2.2, what's the difference between the kernel matrix K with the unbiased estimate of MK-MMD (proposed by Gretton, NIPS 2012, also used in Deep adaptation network, Long, et al. ICML2015)?


Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

After reading through the authors' comments and rereading parts of the submission, I have become a little more positive about this paper.
I am still unsure about the contributions to the ICLR community. The authors merely state "We believe our contributions to ICLR community are clear and valuable" without backing up this claim with an argument.
But in the rest of the authors' comment, they make some good points. I think those points should be made more prominently in the paper itself. I would suggest that the authors describe their approach as using different, complementary encoders of the input sentence and consensus maximization. If they wish to describe this as multi-view learning, that's fine, but I think using the term "consensus maximization" (or something more descriptive like that) in prominent places would be helpful.
If the approach is applicable beyond sentence embedding learning, then it would behoove the authors to describe the approach in a general way so that readers will see how they can apply it to their own tasks. As currently written, the paper is very much focused on sentence embedding learning, which causes me to think that the paper is more appropriate for an NLP venue. But it is true that ICLR publishes papers that are application-specific, so I can't consider this to be a deal-breaker for the paper.
I raised my score to a 6.
--------------------------------- original review follows: ----------------------------------
This paper describes experiments in learning sentence embeddings from unlabeled text. The paper compares a few different compositional architectures and training objectives. The story of the paper focuses on the training of multiple architectures jointly for a single sentence, then ensembling those architectures at test time to represent sentences. One architecture is an RNN and the other is a word averaging model, and the idea is that these two architectures capture different "views" of the sentence.
Pros:
As a general-purpose method to get sentence embeddings without using any resources other than unannotated text documents, this approach has strong results, including solid performance on the SentEval tasks and relatively-low training times.
It was also nice to see how the results depend on the domain of the training data. Review data definitely helps on the several sentiment-related tasks, which provides further evidence of a worrisome aspect of SentEval.
Cons:
Overall, the paper feels incremental and is likely a better fit for an NLP conference. What are the generalizable contributions to the ICLR community? Given the known differences between RNNs and word averaging models for sentences (especially on the SentEval tasks, which, as the authors note, was discussed by Hill et al.), it's entirely unsurprising that combining the two would be a good idea. But even if this were not the case, the ubiquity of ensembles outperforming single models in deep learning also makes it unsurprising that combining these two kinds of model architectures would be beneficial. So I'm just not sure if there is a significant contribution beyond the NLP results. These results seem solid (though a bit incremental), but if the primary contribution is empirical, then the paper would be a better fit for an NLP venue.
In addition, I'm not sure if "multi-view" is an appropriate description of the approach. In Sec. 1, we find the sentence "Compared to earlier work on multi-view learning (de Sa, 1993; Dhillon et al., 2011) that takes data from various sources or splits data into disjoint populations, our framework processes the exact same input data in two distinctive ways."  Therefore, maybe it's not quite accurate to describe this approach using the term "multi-view learning"? I think it would make more sense to use a different term rather than stretch the definition for a well-known one.
I kept expecting the paper to present results when combining the generative and discriminative objectives, but as far as I can tell, this was never done. What would happen if one were to use multitask learning and just optimize the sum of the two losses?
I'd suggest citing and comparing to the results from "Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning" by Subramanian et al. (ICLR 2018) and the Byte mLSTM from "Learning to generate reviews and discovering sentiment" by Radford et al.
I'm not sure how excited we should get about not using any annotations or structured resources for learning sentence embeddings. The authors do not motivate this goal.
Below are more specific comments/questions:
Sec. 2.2 contains the sentence "Ideally, the inverse of h should be easy to compute so that during testing we can set g = h^-1." At this point in the paper, it is not clear what g is going to be applied to at test time, since presumably the following sentence is not going to be available at test time, right? I think it would be good to discuss how the model is going to be used at test time before discussing the inverse of h.
Sec. 2.3:
In Eq. (3), why does the denominator sum always start at 1 no matter what i and j are? That is, why would the denominator always sum over the first N sentences in the dataset?
I think the pooling methods in Table 2 should be described in Section 2.
In Table 2, it is not clear what h_i^{M_i} is. If M_i is the number of words in sentence i, that should be mentioned somewhere.
Sec. 3.1:
"For a given sentence input s with M words, suggested by (Pennington et al., 2014; Levy et al., 2015), the representation is calculated as z = (\hat{z}_f + \hat{z}_g) / 2, where \hat{z} refers to the post-processed and normalised vector, and is mentioned in Table 2."  I don't understand. Where is this mentioned in Table 2?
Minor issues follow:
Sec. 1:
"Distributional hypothesis" --> "the distributional hypothesis"
"in machine learning community" --> "in the machine learning community"
"and distributional hypothesis" --> "and the distributional hypothesis"
"the linear/log-linear models" --> "linear/log-linear models"
"based on distributional hypothesis" --> "based on the distributional hypothesis"
"contraint" --> "constraint"
Sec. 2:
"marry RNN-based sentence encoder" --> "marry RNN-based sentence encoders" or "marry the RNN-based sentence encoder"
Sec. 2.1:
"only hidden state" --> "only the hidden state"
Sec. 2.2:
"prior work with generative objective" --> "prior work with generative objectives"
Sec. 2.3:
"with discriminative objective learns" --> "with the discriminative objective learns"
Sec. 3.1:
In Table 3, I don't see where superscript 5 is shown in the table.
Sec. 3.2:
In Table 5, the numbered superscripts at the top of the table do not show up next to the methods in the actual table rows.


We introduce HyperGAN, a generative network that learns to generate all the weight parameters of deep neural networks. HyperGAN first transforms low dimensional noise into a latent space, which can be sampled from to obtain diverse, performant sets of parameters for a target architecture. We utilize an architecture that bears resemblance to generative adversarial networks, but we evaluate the likelihood of samples with a classification loss. This is equivalent to minimizing the KL-divergence between the generated network parameter distribution and an unknown true parameter distribution. We apply HyperGAN to classification, showing that HyperGAN can learn to generate parameters which solve the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters. We also show that HyperGAN can also provide better uncertainty than standard ensembles. This is evaluated by the ability of HyperGAN-generated ensembles to detect out of distribution data as well as adversarial examples. We see that in addition to being highly accurate on inlier data, HyperGAN can provide reasonable uncertainty estimates.

This paper proposes a technique for learning a distribution over parameters of a neural network such that samples from the distribution correspond to performant networks. The approach effectively encourages sampled parameters to have low loss on the training set, and also uses an adversarial loss to encourage the distribution of parameters to be Gaussian distributed. This approach can improve performance slightly by using ensembling and can be useful for uncertainty estimates for out-of-distribution examples. The approach is tested on a few simple problems and is shown to work well.
I am definitely in favor of exploring adversarial divergences (using a critic as a differentiable loss to compare two distributions) in unusual settings, and this paper certainly does this. The idea of transforming samples from a prior such that the transformed sample corresponds to useful network parameters is interesting. The results also seem promising. However, currently the mathematical description of this method is completely unclear and ridden with many errors. I can understand at a reasonable level what the approach is doing from Figure 1, but the definitions and equations given in Equation 3 are at times nearly incomprehensible "mathiness". I'm giving the paper a borderline accept because the idea is interesting and the results are OK; I will raise my score if Section 3 is dramatically improved. I give some specific examples of issues with Section 3 in my specific comments below. I'd also note that the paper does a somewhat poor job comparing to existing work - only section 4.2 includes a comparison to existing "uncertainty" methods. This should also be improved - the authors should implement the existing methods and use them as a point of comparison in all of their experiments. As a final high-level note, the approach is described at various points as an "autoencoder" particularly in reference to the adversarial autoencoder. However, the approach does not "autoencode" anything - there is no reconstruction term, or input apart from the noise samples. The only thing it has in common with the adversarial autoencoder is the use of a critic to enforce a distributional constraint. Calling it, or comparing it to, an autoencoder is confusing and misleading.
Specific comments:
- You mention fast weights in related work. I believe Hinton and Plaut were the first to propose fast weights in "Using Fast Weights to Deblur Old Memories", and I'd also suggest mentioning "Using Fast Weights to Attend to the Recent Past" which is a more recent demonstration that fast weights can be useful on modern problems.
- The are some issues with your description of Equation 1: First, I don't believe you define G(z) (I assume it is the "decoder" network; please define it). Second, in practice I don't believe you actually use JSD or MMD for D_z; you use a critic architecture which in some limit approximates some statistical divergence but in practice they typically don't (see e.g. Arjovsky and Bottou 2017; Fedus et al. 2017; Rosca et al. 2018). Third, writing Q_z \sim Q(z | x) seems strange to me - Q_z is a distribution, and I don't believe that Q(z | x) is a distribution over distributions, so how are you sampling a distribution (Q_z) from Q(z | x) as suggested by the use of the \sim notation? I think you simply mean that Q_z is Q(z | x) approximately marginalized over x.
- Equation 2 is also not clear. First, the sentence before starts "Suppose the real parameters \theta^* \sim \Theta..." The equation itself does not include \theta^* or \Theta so I don't see what this is referring to. Second, the expression for an m-dimensional is written \mathcal{N}(0, \sigma^2, I_m). It's not clear why there is a comma before I_m, and I_m is not defined (though I assume it is the m \times m identity matrix) - did you mean to multiply I_m by \sigma^2? Third, it looks like you actually define P_z twice, once as "an m-dimensional isotropic Gaussian" and again as "a Kd-dimensional isotropic Gaussian"; am I to infer that m = Kd? Why use both? Fourth, you mention the joint P(x, y) but the expectation is taken over P_x and P(y | x). Why call it P_x and not P(x)? And why not compute the expectation over P(x, y)? Fifth, you write "Here the encoder..." -- you never define that Q(z) or G is "the encoder", I assume Q(z). It is strange to take the expectation over Q(z) (I assume sampling z \sim Q(z)) but then have the term Q(z) appear in (2). How are Q(z) and Q_z related? On that note, I don't see how (2) is an autoencoder, since there is no Q(z | x) term. It appears instead that you are sampling z from Q(z) which doesn't condition on x. So what is being autoencoded? Related, you write "all the q_k (that will generate different layers) will be correlated, unlike dimensions of z which are drawn to be independent from each other." But if Q(z) = [q_1, ..., q_K] then doesn't the secont term in (2) suggest that they are being enforced to be similar to the prior P_z, and therefore uncorrelated? Note that you also say later on "The job of the regularizer D_z(P_z, Q_z) is to force each embedding q_n to approximate P_z." Frankly at this point I will stop pointing out issues with this equation and discussion since they are so widespread.
- In your definition of your ensemble scoring rule, you are taking the sum over N + 1 elements (n = 0 to N) but dividing by N.
- In 4.2, do you use the same model architecture/training/regularization etc. as in previous studies? If not I think comparing the different methods will be conflated by differences in training procedures. Since you do not report results in many experimental settings, I assume you don't.
- In Figure 3, why not plot the true standard deviation around the true function? It appears you are only plotting +/- 3 stndard deviations for the learned function.
- Why not include 100 models L2 on Figure 3?
- It's not clear to me why you define your "disagreement d" when it appears the same as the entropy score you used in 4.4.
- A stronger and more convincing attack would be to attack the ensemble of models, instead of attacking a single model and testing on the ensemble.


Machine learning classifiers are often trained to recognize a set of pre-defined classes. However, in many real applications, it is often desirable to have the flexibility of learning additional concepts, without re-training on the full training set. This paper addresses this problem, incremental few-shot learning, where a regular classification network has already been trained to recognize a set of base classes; and several extra novel classes are being considered, each with only a few labeled examples. After learning the novel classes, the model is then evaluated on the overall performance of both base and novel classes. To this end, we propose a meta-learning model, the Attention Attractor Network, which regularizes the learning of novel classes. In each episode, we train a set of new weights to recognize novel classes until they converge, and we show that the technique of recurrent back-propagation can back-propagate through the optimization process and facilitate the learning of the attractor network regularizer. We demonstrate that the learned attractor network can recognize novel classes while remembering old classes without the need to review the original training set, outperforming baselines that do not rely on an iterative optimization process.

The paper addresses the incremental few-shot learning problem where a model starts with base network and then introduces the novel classes, building a connection between novel and base classes via an attention module.
Strengths:
+ clear writing.
+ the experiments are compared with related work and the ablation studies can verify the effectiveness of the proposed (or "introduced" would be a precise term) recurrent BP.
Weakness:
- [Novelty]
The paper title is called attention attractor network, which shares very relevance to previous CVPR work (Gidaris & Komodakis, 2018). So the first thing I was looking for is the clear description of the difference between these two. Unfortunately, in related work, authors mention the CVPR work without stating the difference (last few lines in Section 2). As such, I don't see much novelty in the paper compared with previous work. Eqn. (7)-(10) explicitly describes the attention formula. What's the distinction from the CVPR work?
- [Motivation of the regularizer using Recurrent BP is not clear]
The use of recurrent BP is probably the most distinction from previous work. However, I don't see a clear description on why such a technique is necessary.
Starting from the first line in Section 3.3, "since there is no closed-form of the regularizer in Eqn (13)", E needs BPTT or the introduced recurrent BP. This part is simply a re-adaption of other algorithms. A very simple question is, how about use other regularizers to replace Eqn (13)?
- [Some experiments missing]
The experiments section 4.6 uses a case of None and "best WD" to address some of my concerns. This is good. Does the "gamma random" indicates only E is used without the ||W||^2? why the best WD for one-shot is zero? This implies the model is best for applying no weight decay?
What's the effect of using the recurrent BP technique to the CVPR work? Is there some similar improvement? If yes, then the paper makes some contribution by the regularization. If not, what's the reason?
How about using the truncated BPTT with a larger T?
In general, I think the recurrent BP part should be the highlight of the paper and yet authors fail to spread such a spirit in the abstract or title. And there are some experiments missed as I mentioned above.


Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint. Most previous works focus on the case with a single manifold. However, in practice it is quite common that the optimization problem involves more than one constraints, (each constraint corresponding to one manifold). It is not clear in general how to optimize on multiple manifolds effectively and provably especially when the intersection of multiple manifolds is not a manifold or cannot be easily calculated. We propose a unified algorithm framework to handle the optimization on multiple manifolds. Specifically, we integrate information from multiple manifolds and move along an ensemble direction by viewing the information from each manifold as a drift and adding them together. We prove the convergence properties of the proposed algorithms. We also apply the algorithms into training neural network with batch normalization layers and achieve preferable empirical results.

The title is misleading, since only two particular manifolds are studied in this work. In addition, the proposed methods cannot be applied to a larger or a general class of manifolds. Therefore, you should update the title.
There are multiple problem definitions proposed in the paper. They are not compatible with each other and also with the proposed methods. In addition, some of the proposed problem definitions are incorrect, as explained below:
You should be more precise about the definition of the manifold you consider in this paper. For example, in equation (1), please define your manifold of interest more precisely checking some standard textbooks.
Please define intersection of manifolds, what do you mean by which intersection of which type of manifolds?
In the contribution (1); the paper does not introduce an algorithm to deal with optimization on with multiple manifolds, but for a particular type of individual manifolds.
In the contribution (2): It is not clear why and how the proposed method can be applied to optimization on manifolds with momentum (what do you mean by use of momentum here?), and regularization (what do you mean by regularization?). There are many problems with this claim, but you can simply consider that applying momentum and regularization will affect the geometry of loss landscape.
Definition of retraction is not precise, please fix it.
What is L in equation (2)?
Please define neighborhood in U_x in Lemma 2.1.
What is || ||in Lemma 2.1?
As you also noticed on page 3, an intersection of manifolds may not be a manifold. Then, your proposed first problem (1) fails. Therefore, you should completely change your claims on your problem definitions and contributions.
What do you mean by “We add a drift which contains information from the other manifold to the original gradient descent on manifold”? What is “the information from the manifold”? In equations (3) and (4), you just apply optimization on manifolds individually.
How do you compute/determine a_k^(1) and a_k^(2)? How do they affect the theoretical and experimental results?
In your claim “From the construction of bk, we can see that the smaller the correlation between gradf(xk) and hk is, the smaller effect the information from M2 brings”, it is not clear how “the information from M2” affects? First, again, what is “the information”? Second, b_k^(1) and b_k^(2) are computed for individual manifolds separately. Then, how “the information” make an effect?
 In Theorem 2.2, what do you mean by “then xk convergence to a local minimizer“?
What is <,> in Theorem 2.2?
What is ^ in Theorem 2.3?
What is v in proof 6?
What is an engine value?
What does P (1) xk gradf(yk) denote in computation of h_k? For example, gradf(yk) is a vector on tangent space of the second manifold at yk. Then, how do you project orthogonally this projected vector to the tangent space of the first manifold at xk?
They may be completely different geometries, and such an “orthogonal projection” may not exist in general. Then, how do you compute and calculate that projection?
All the theoretical results given in the paper are not about convergence of parameters on a manifold at the intersection or product of manifolds but for an individual manifold. For example, x and y belong to manifolds M1 and M2, and convergence results is about x. How are they related to parameters at the intersection or product of manifolds?
The statements regarding batch normalization are confusing and also sound incorrect:
Do you apply batch normalization on weights on BN(w)?
Please explain what you mean by “BN(w) has same image space on G(1, n) and St(n, 1)“. There are not such results in the papers Cho & Lee (2017); Huang et al. (2017) you cited for these results.
What do you mean by “applying optimization on manifold to batch normalization problem”?
In your statement “However, the property of these two manifold implies that we can actually optimize on the intersection of two manifolds”. Please explain how does this property imply this result more precisely?
Please define “Grassmann manifold G(1, n)“ more precisely. In your notation, together with explanation of the notation for St(n,p), G(1,n) is like a set of 1xn dimensional row vectors, while St(n,1) is an nx1 dimensional column vector, Then, their intersection is an empty set and your proposal for optimization on a vector on their intersection is wrong.
Notation and definitions used in (9) are wrong and confusing. Please check and revise them.
In the whole paper, the problem, method, solutions, theorems, and contributions are proposed for optimization using parameters which belong to intersection of some manifolds. Then, suddenly, you start considering optimization on product manifolds, and give the results for that;
What does the statement “Then we apply Algorithm 1 to update parameters, which means we optimize on a product manifold” mean?
What do “G(1, k1) × · · · G(1, kn)” and “St(k1, 1) × · · · St(kn, 1)” denote?
Don’t you perform optimization on intersection of manifolds? Why do you ignore your original problem and methods, and consider this problem?
In addition, how do you use your Algorithm 1 for optimization on product manifolds? Optimization on intersection on manifolds and product manifolds are completely different problems. If they are same or related to each in particular cases in your specific definitions, then you should provide these definitions more precisely.
What do you mean by optimization on product manifold of weights of all layers? If you compute a product manifold for spaces of all layers, then you simply perform a shallow optimization on a huge matrix containing millions of dimensions according to this definition. First, how do you do that? Second, how can you train a large network using this approach?
In the experiments, please first give variance of errors. These results are statistically insignificant.
Which problem is solved to perform these experiments is not also clear (see above).
The results reported in the paper are also not good, may be due to the mathematical and algorithmic  problems and errors mentioned above. Please clarify them, and provide additional results, especially using other datasets (small scale mnist and large scale imagenet), and networks (mlp, vgg, resnet etc.)
Related work is also incomplete, such that many traditional and recent work on optimization on multiple manifolds are omitted.