A common strategy in modern learning systems is to learn a representation which is useful for many tasks, a.k.a. representation learning. We study this strategy in the imitation learning setting for Markov decision processes (MDPs) where multiple experts' trajectories are available. We formulate representation learning as a bi-level optimization problem where the "outer" optimization tries to learn the joint representation and the "inner" optimization encodes the imitation learning setup and tries to learn task-specific parameters. We instantiate this framework for the imitation learning settings of behavior cloning and observation-alone. Theoretically, we provably show using our framework that representation learning can reduce the sample complexity of imitation learning in both settings. We also provide proof-of-concept experiments to verify our theoretical findings.

This paper theoretically explores reinforcement/imitation learning via representation learning. The key theoretical question being investigated is the relationship between representation learning in a multi-task/meta learning setup and its dependence to the sample/task complexity. The paper sets up the problem in bilevel optimization framework, where the inner optimization learns/optimizes task specific losses, while the outer optimization learns the representation used in the inner level tasks. The main takeaway from the two theorems (which are the core contributions of this paper) are that when the number of tasks is higher than the number of samples, representation learning can reduce the sample complexity. The paper explores two scenarios in imitation learning, namely behavioral cloning and when only the states of the experts are available (and not their actions). Some experiments are provided to empirically validate the theory.
Pros:
1. The paper presents a theoretical investigation into multi-task/meta learning for RL via learning representations. While, the main theoretical contributions are perhaps marginal with regard to prior work, the problem setting (RL) seems novel and the two theorems in this context are interesting.
2. The paper is well-written and appears to be very rigorous. I did not check for the correctness of all technical parts. There are several "abuse of notations" in the main text, which sometimes impact the otherwise smooth read of the technical parts.
3. A good and concise review of RL concepts is provided.
Cons:
1. The paper lacks a good literature review to place this work in the right context. For example, while the paper refers to the works of Maurer 2016 and Sun et al. 2019 at several places, it is not formally and clearly mentioned anywhere what are the similarities to these prior works and what are the new contributions. For example, Maurer 2016 proposes a multi-task learning setup using representation learning, and most of Theorem 4.1 in this paper is taken from the results in that paper. While, the current paper uses bilevel optimization setting in an RL context, it is not clear to me if this (bilevel + RL) setting has any significant bearing against the theoretical results furnished by Maurer 2016. For example, the theorems in this paper (as far as I see) show that the bounds are scaled by a constant defined by H^2, the trajectory length? If there is something beyond this, then the paper needs to explicitly point it out. The same comment goes with the results against Sun et al. 2019 in Theorem 5.1.
2. Given that the main goal of this paper is to connect sample complexity with representation learning, it is important the paper provide a theorem stating this precisely. Theorems 4.1 and 5.1 provide a general bound, and the sample complexity is being described in the explanations of this theorem, which is very informal. Also, against what is claimed in the abstract, it appears that representation learning helps reduce sample complexity only when the number of tasks are larger, which perhaps needs to be explicitly mentioned. Also, note that there is a bearing of the bound on the trajectory length H (Theorem 4.1, and 5.1). Shouldn't this factor be also accounted for when explaining the sample complexity?
3.  There has been several recent works on model-agnostic meta-learning (that also uses bilevel optimization and implicit gradients), however, older works on meta-learning (only for imitation learning) have been cited. The paper should include more recent works in this area and contrast against their theoretical findings.
Apart from these, below are some minor comments that could help improve the reading of this paper:
a. Theorem 3.1 is not really a theorem, since it is very informal. Also, fix the Theorem numbers.
b. Bullet 1. after Theorem 3.1, \ell^x(\pi) concentrates to \ell^x(\pu*). Also, the mention about sample complexity here should be backed with some reference/citation.
c. Assumption 4.2: The notation \pi_\mu(s)_{\pi*(\mu(s)) is unclear, shouldn't the subscript contain an argmax over the actions for \pi*?
d. Theorem 4.1 and 5.1, what does it mean by "probability 1-\delta over the choice of the dataset X" ? Also, \mu^n seems undefined.
e. The first two terms in Theorem 4.1 are claimed standard, provide the citations?
f. Theorem 4.1, perhaps use some other notation for c, which is defined as the cost/reward in the RL setting.
Overall, the paper has some interesting theoretical results, and is mathematically rigorous, however lacks a clear distinction from prior and more recent works in this area.


Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost. To make CNNs more efficient, many methods have been proposed to either design lightweight networks or compress models. Although some efficient network structures have been proposed, such as MobileNet or ShuffleNet, we find that there still exists redundant information between convolution kernels. To address this issue, we propose a novel dynamic convolution method named DyNet in this paper, which can adaptively generate convolution kernels based on image contents. To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs. The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged. Specifically, for ShuffleNetV2 (1.0), MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 37.0%, 54.7%, 67.2% and 71.3% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1.0%, −0.27%, −0.6% and −0.08%. Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87×, 1.32× and 1.48× on CPU platform respectively. To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.

Main contribution of the paper
- The paper proposes a dynamic convolution selection method can be applied to arbitrary classification networks based on the global average pooled (GAP) feature map info.
- The method obtained improvements over various networks (SuffleNet v2, MobileNet v2, ResNet 18) on ImageNet.
Methods
- Given the set of fixed convolutional filters, the method dynamically selects the (weighted sum) kernels by given a kind of channel attention.
- The GAP of the features gives the channel attention on each stage, and the method applies the dynamic selection of the kernels.
- The number of channels in skip-connection shluld be the same because it should be elementwise multiplied with the channel attention acquired from GAP.
- The author slightly revises the baseline networks to set the networks integrated with the proposed method to have smaller Flops.
Questions
- According to Figure 4, it seems that the proposed add-on requires many parameters because it would include a FC layer for each block. But we cannot find the number of parameters in this paper.
- The parameter $g_t$ is defined as 6. The experiment shows the ablation to the case $g_t$ =1, but what if we set the parameters to other numbers?
Strong points
- The proposed model achieved improvement with fewer Flops on large scale image classification dataset.
- The method shows effectiveness when it is attached to various classification networks.
Concerns
- The main concern of the reviewer is that the model shares the core contribution to the existing method; squeeze-and-excitation network (SEnet, Hu et.al.). The method also proposes the attention-based scaling of channels, where the attention comes from GAP, so the reviewer thinks that it is possible to explain this work as some variation of SEnet.
The author should clarify the difference and the strong points of the proposed block compared to SEnet.
- Also, the reviewer cannot guarantee that the networks trained by the proposed method can transfer the knowledge to other tasks such as detection.
The reviewer thinks that it is a critical part because one of the primal reasons for training the network is to use them as the pre-trained backbone for the other tasks.
Regarding this, the baseline methods (MobileNet V2, Shufflenet v2, ResNet)  are widely used as a pre-trained backbone for object detection, and the papers mention the CoCo object detection results using the pre-trained backbones from their method. The reviewer thinks that the experiment regarding this should be included.
- The other thing is that the parameter increases. As in the question, the reviewer thinks that the number of parameters would be increased. The reviewer agrees that some recent works focus more on Flops, but the number of parameters is also discussed in general, when telling about the 'model size'.
Conclusion
- The author proposed a dynamic kernel selection method (add-on), which can enhance the classification accuracy of the baseline network.
- However, the reviewer cannot convince the novelty of the proposed approach and usefulness of the pre-trained backbone network from the proposed method when applying it to the other tasks (Object detection).
Inquiries
- Clarifying the difference between SEnet.
- Testing the ImageNet trained network of the proposed method into an object detection task (as the pre-trained backbone).
- Discussing the number of the parameter as well.


As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform parallel model training.  describe two variants of data-parallel SGD that quantize and encode gradients to lessen communication costs. For the first variant, QSGD, they provide strong theoretical guarantees. For the second variant, which we call QSGDinf, they demonstrate impressive empirical gains for distributed training of large neural networks. Building on their work, we propose an alternative scheme for quantizing gradients and show that it yields stronger theoretical guarantees than exist for QSGD while matching the empirical performance of QSGDinf.

The authors propose a new scheme for quantizing gradients which are followed by the previous work QSGD [1]. They show that it yields stronger theoretical guarantees than QSGD while showing a great empirical performance.
The main difference between their scheme NUQSGD and QSGD is that they use nonuniform quantization (0, 1/2^{s},  …., 2^{s-1}/2^{s}, 1) instead of uniform quantization (0, 1/s, …, (s-1)/s,1).  Intuitively, by the way, it could reduce quantization error and variance by better matching the properties of normalized vectors.
The results are in 2 parts. First comparing with QSGD, they establish stronger convergence guarantees for NUQSGD, under standard assumptions. They also establish theoretical results for the variance upper bound and expected communication cost of their scheme. Second, they show strong empirical performance on deep models and a large dataset, with an efficient implementation in PyTorch.
However, there are several issues and questions that if fixed or illustrated could be a great paper.
	1) The author claim NUQSGD achieves stronger convergence guarantees comparing with QSGD but hasn't illustrated the point in detail. On page 6, the paragraph named 'NUQSGD vs QSGD' mainly claims that variance upper bound controls the guarantee on the convergence speed by empirically showing the results of variance upper bound. It would be great to include more theoretical analysis which demonstrates the importance of variance upper bound for convergence speed guarantee.
	2) In the experimental part, they control the hyperparameters including batch-size, base learning rate, momentum, and weight decay to be identical with each method. This may cause tuning biases (the setting may favor one method but hurt others' performance).
	3) Although the paper mainly focuses on comparing with QSGD, there are several relative communication efficient training algorithms which I think are worth to compare empirically (at least one of them). For example:
		a. Deep Gradient compression [2]
		b. signSGD [3]
		c. TernGrad [4]
	4) In figure 4, the encoding cost is significantly increased from 4-bit to 8-bit NUQSGD. Any reason why it happens? Is it due to inefficient encoding implementation?
I agree with the authors' point that it's worth to explore the interaction between NUQSGD with more complex reduction patterns like ring-based. Since the ring-based algorithm like all-reduce is more popular in practice nowadays, interacting with it would have a better practical meaning.
[1] D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efﬁcient SGD via gradient quantization and encoding. In Proc. Advances in Neural Information Processing Systems (NIPS), 2017.
[2] Lin Y, Han S, Mao H, Wang Y, Dally WJ. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887. 2017 Dec 5.
[3] Bernstein J, Zhao J, Azizzadenesheli K, Anandkumar A. signSGD with majority vote is communication efficient and fault-tolerant. arXiv. 2018 Oct 11.
[4] W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. TernGrad: Ternary gradients to reduce communication in distributed deep learning. In Proc. Advances in Neural Information Processing Systems (NIPS), 2017.


We have rigorously proved the existence of a group algebraic structure hidden in relational knowledge embedding problems, which suggests that a group-based embedding framework is essential for designing embedding models. Our theoretical analysis explores merely the intrinsic property of the embedding problem itself hence is model independent. Using the proposed framework, one could construct embedding models that naturally accommodate all possible local graph patterns, which are necessary for reproducing a complete graph from knowledge triplets. We reconstruct many state-of-the-art models from the framework and re-interpret them as embeddings with different groups. Moreover, we also propose new instantiation models using simple continuous non-abelian groups.

I would like to thank the authors for the detailed rebuttal.
My concern is generalizing something that is not correctly describing the object we should model (KGs).
For instance, "all properties mentioned in the paper, including closure, identity, inversion, and associativity, are desired properties of relation embedding spaces, rather than existing properties of specific knowledge graphs. " => should we not care about the actual relations of knowledge graphs? Should we not build a theory that closely matches the properties of the actual objects we are modeling? The argument seems to be: forget the actual knowledge graph, here is a generalization of existing KGE methods. Even the proof provided was unrelated to my question: I gave an example: can you list all relations in FB15K and prove that they fit the group axioms? If they don't fit, what is the advantage of modeling the relations as a group? What is the representation error this causes?
The authors misunderstood my comment about (Bloem-Reddy and Teh 2019). "Therefore the previous works mentioned by the reviewer above are irrelevant." => this is a weird statement. Permutation invariance is relevant to all objects that are graphs. I recommend reading the long history relating graph models and exchangeability (Persi Diaconis has a good overview). I was just mentioning that I don't see why KGs should have another associated permutation group besides permutation invariance. I would also recommend following the emerging literature in graph representation learning using group theory, of which KGs are but a special case.
"The (entity) manifold, however, means that the relations no longer form a group. I see no easy fix." => I clarified it, since I was talking about the entity embedding.
I did not see a revised version of the manuscript.
I will raise my score because the paper could be published as a niche paper. We generalize KGE but we acknowledge its shortcomings: And here is a way to measure the embedding error of modeling the relations as a group, when they are not actually a group.
--------------
The paper is well written and an interesting read. I think it complements well the existing literature.
Unfortunately, I think the paper overstate its claims. It is clear that permutation groups are the natural language of all graphs (Kondor and Trivedi, 2018) and (Bloem-Reddy and Teh, 2019). It is less clear that knowledge graphs also comply with another set of group axioms. Why would the set of relations be closed? Say, Owns \cdot Spouse_of = ? . I cannot see why and how this is necessary. Must all the relations be closed? I understand why this is true for KGE but this is not true for KGs, which only shows that KGEs may not be the right method to represent KGs moving forward. I see no easy fix.
The rest of the paper is straightforward, just applying the definitions of groups. I found the classification of different methods interesting, and should be made more clear in the experiments.
The experimental results are interesting but their significance is unclear. Please add standard deviations to all experiments. We cannot have a sense of the significance of the results without knowing how many runs were executed, how they were executed (e.g., k-fold cross-validation, bootstrapping), and their standard deviation.
I dispute that the paper proves that the definition of groups emerges purely from the nature of the knowledge graph embedding (KGE) task. I would actually say the opposite, it is clear to me that for most KGs, their relations do not fit the definition of a group. I am open to be (formally) proven wrong. E.g., can you list all relations in FB15K and show that they fit the group axioms?
Fixing the paper: Maybe the paper could be rewritten, constrained to Euclidean spaces? Then prove (formally) that the group axioms are a sufficient(?) and necessary(?) condition for such embeddings?
Minor issues:
honestly could not make sense of the abstract:
The sentence choices
-  "which suggests that a group-based embedding framework is essential for model design"
- "Our theoretical analysis explores merely the intrinsic property of the embedding problem itself without introducing extra design"
in the abstract are very strange. It means nothing to a reader at that point. What a model design? What is an extra design? A modeling assumption? A model prior?
"Using the proposed framework, one could construct embedding models that naturally accommodate all possible local graph patterns, which are necessary for reproducing a complete graph from atomic knowledge triplets" => what is an "atomic knowledge triplet"? Why are they necessary to reproduce a complete graph? This is all very confusing.
" contradicts to the entity regularization" =>  contradicts the entity regularization
References:
Kondor, R. and Trivedi, S., 2018. On the generalization of equivariance and convolution in neural networks to the action of compact groups. ICML
Bloem-Reddy, B. and Teh, Y.W., 2019. Probabilistic symmetry and invariant neural networks. arXiv:1901.06082.


Relational reasoning-the ability to model interactions and relations between objectsis valuable for robust multi-object tracking and pivotal for trajectory prediction. In this paper we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning. We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment. We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs. To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature.

The paper presents a class-agnostic method for tracking multiple moving objects (MOHART) that extends an existing single-object tracking method (Hierarchical Attentive Recurrent Tracking, HART). Similarly to HART, MOHART utilizes an attention mechanism and LSTM units. The extension form HART to MOHART is done in two main steps: HART is applied to multiple objects in parallel, with a presence variable attached to each, and a permutation-invariant network that learns the interactions between the objects.
The method is tested in two groups of experiments: synthetic data of moving circles, and on several real datasets. The results on the moving circles are enlightening. The paper shows that although MOHART outperforms HART in all circumstances, if the forces between the circles change randomly every number of steps, MOHART outperforms HART by a very large margin. This is because the forces between the circles are determined by their color, which is observable by the network, and MOHART, which capable of learning their interaction, learns the pattern of changing forces. HART, being an independent-particle model, cannot learn the interactions.
The results on the real datasets demonstrate superiority over HART, and in a series of ablation experiments, the Authors are able to show that MOHART's ability to utilize interactions between objects is responsible for the improvement. For example, the improvement over HART is greater when egomotion is significant, because egomotion creates correlations between the movement of objects in the camera frame, which MOHART can utilize.
I found the paper enlightening, based on a neat idea. The experimental results are extensively analyzed and the Author's insights about their algorithm are substantiated by the experiments.
However the difficulty I had with the paper is that the results are only compared to HART. HART was published in 2017, and the field of scene understanding and tracking had been addressed in multiple papers since then. It is one of the more competitive and application-driven fields in computer vision. Comparing MOHART only to its sister-method from over two years ago significantly limits the usefulness of paper IMHO. The expectation IMHO is not that MOHART outperform each and every method out there, but that the reader know where MOHART stands compared to other methods (e. g. tracking by detection).
I will refrain from pointing to specific papers to compare to, because I believe the Authors should chose the settings, datasets, criteria, and metrics that are the most convenient for them for comparison. The papers surveyed in Section 2, especially under "Tracking-by-Detection", have many followups. Recent papers from those that claim state of the art could good candidates for comparison.
In view of the above, I am inclined at this stage to reject the paper.


We propose an efficient inference method for switching nonlinear dynamical systems. The key idea is to learn an inference network which can be used as a proposal distribution for the continuous latent variables, while performing exact marginalization of the discrete latent variables. This allows us to use the reparameterization trick, and apply end-to-end training with stochastic gradient descent. We show that the proposed method can successfully segment time series data (including videos) into meaningful "regimes", by using the piece-wise nonlinear dynamics.

Thank you for an interesting read.
As far as I understand, the paper claims two contributions:
1. A combination of collapsed variational inference and amortised inference for SNLDS, to make the training pipeline fully differentiable;
2. An improved loss function upon the variational lowerbound (ELBO) to force the model to use the discrete states.
======= novelty =======
The 1st idea is combinatorial:
1. The forward-backward algorithm is a standard inference method for HMM-like sequence models; collapsed variational inference has been investigated extensively in 2000s when hierarchical Bayes models were actively developed; amortised inference is widely used in variational auto-encoders.
2. The combination of the above two inference methods on S(N)LDS is new to the best of my knowledge. However, this combination has been proposed on a similar model called Kalman VAE ( Fraccaro et al. 2017), where the sequence model can be viewed as a "soft" version of SLDS.
The 2nd idea is interesting but not very well explained to some extent:
1. The goal of the modified objective function is to encourage the model to use the discrete states (instead of pushing all useful information to the continuous states). It is interesting as it regularises the *exact posterior* of the discrete states conditioned on the *approximately inferred* continuous states.
2. I believe the entropy regulariser is non-differentiable as it is based on a *histogram* estimate of the temporally averaged discrete state distribution. How exactly is this regulariser implemented?
3. I agree adding the KL regulariser can avoid the iterating assignment pathology, however, is random assignment of the regime preferred in any case? From the introductory example, I think contiguous segments are preferred.
======= significance =======
Experiments consider 3 synthetic examples for sequence segmentation (so that ground truth is available). The proposed approach performs significantly better which is a good sign. The paper also provides useful analysis on the effects of balancing parameter tempering which is always welcome.
However, two baselines are missing:
1. To claim the significance of the collapsed variational inference approach, a non-collapsed inference version of the proposed SNLDS model needs to be compared. The authors did discuss this and mentioned possible workarounds (e.g. using the Gumbel-softmax trick for discrete state inference), but the comparison is not reported. If compared, this will serve well as an ablation study for the inference method.
2. The paper also provides comparisons across models, but I do think the Kalman VAE model needs to be compared to SNLDS. Both models are more flexible than the original SLDS, but the complexity is added in different ways. Since I think the inference mechanisms are similar (both using forward-backward inference for top-level latents and amortised inference for bottom-level latents), this comparison would provide a better ablation study on the modelling side.
======= clarity =======
1. The paper presentation is overall clear to me, although I found the many sentences in parenthesis a bit distracted, so I would suggest maybe using footnotes for them instead.
2. For readers who are less familiar with HMMs/forward-backward algorithms, the papers can be difficult to understand, as it skips all the detailed computation of the gamma terms. I would suggest adding the details in the appendix, and/or visualise the intuition using e.g. message passing on factor graphs.
3. I found the related work well presented with most relevant papers, although I do think the Kalman VAE approach is highly relevant which needs to be cited and discussed.
======== references ========
Fraccaro et al. (2017). A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. NeurIPS 2017


Meta reinforcement learning (meta-RL) is able to accelerate the acquisition of new tasks by learning from past experience. Current meta-RL methods usually learn to adapt to new tasks by directly optimizing the parameters of policies over primitive actions. However, for complex tasks which requires sophisticated control strategies, it would be quite inefficient to directly learn such a meta-policy. Moreover, this problem can become more severe and even fail in spare reward settings, which is quite common in practice. To this end, we propose a new meta-RL algorithm called meta goal-generation for hierarchical RL (MGHRL) by leveraging hierarchical actor-critic framework. Instead of directly generate policies over primitive actions for new tasks, MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals as independent RL subtasks. Our empirical results on several challenging simulated robotics environments show that our method enables more efficient and effective meta-learning from past experience and outperforms state-of-the-art meta-RL and Hierarchical-RL methods in sparse reward settings.

This paper studies the problem of leveraging past experience to quickly solve new control tasks. The starting point (and perhaps the main contribution) is the observation that some tasks have similar high-level goals, while differing in how those goals are achieved. To that end, the paper introduces an meta-RL algorithm that, given a new task, attempts to solve it by adapting a high-level, goal-setting module, and learn a new, low-level policy to reach each commanded goal. The proposed method might be viewed as a combination of PEARL [Rakelly 19] and HAC [Levy 19]. The proposed method is compared against state-of-the-art hierarchical RL and meta-RL methods on four robotic manipulation tasks. The proposed method outperforms the baselines on each task.
While the proposed method is quite strong empirically, I am leaning towards rejecting this paper because many of the claims made in the paper are not empirically validated. While much emphasis is put on the hierarchical aspect of the algorithm, I don't think that the tasks used in the experiments require hierarchy to solve (see [Plappert 18]). In the introduction, the second claim is that the proposed method "focus[es] on meta learning the overall strategy … [and] provides a simpler and better way for meta RL." While the experiments show that the proposed method learns better than baselines, I don't think the paper show that the proposed method learns some sort of "overall strategy." I don't think that the proposed method is simpler than the baselines.
A second concern is that I'm confused about the experimental protocol. If the high-level policy outputs a desired XYZ position for the gripper (Section 5.1), how can the high-level policy indicate when the gripper should be closed to pick up the block? How is the reward function for the low-level policy defined? PEARL doesn't have access to this extra information (the reward function), right?
A third concern is the large number of grammatical errors in the paper.
I would consider increasing my review if (1) a new plot were added to visualize the commanded subgoals (I have a hunch that the high-level policy directly outputs the true goal, obviating the need for hierarchy and contradicting the claim that the method "learns to generate high-level meta-strategies over subgoals"); (2) the experimental protocol were clarified; and (3) the number of grammatical errors were significantly reduced.
Other comments:
* "inefficient to to directly learn such a meta policy" -- Why? Also, "to to" is repeated.
* "Deep Reinforcement learning" -- "Reinforcement" shouldn't be capitalized.
* "failing to generalize" -- Can you add a citation?
* "it would be quite inefficient to directly learn such a policy…": Doesn't [Plappert 18] do exactly this?
* "When the tasks distribution is much wider … these methods can hardly be effective…" -- Where is this claim substantiated? Also, "tasks" should be singular.
* "sparse reward settings which is" -> "sparse reward settings, which are"
* "the above mentioned problems" -> "the problems mentioned above"
* "our algorithm focus on meta learning … which provides a much simpler …" -- Where is it shown that the proposed method is simpler? Also, "focus" should be "focuses"
* "1991),which" -- Missing space
* "complex tasks which requires" -> "complex tasks that require"
* "Nachum et al … set of sub-policies" -- Run on sentence.
* "... human leverage…" -- Don't humans also transfer low-level knowledge across tasks, in addition to high-level knowledge? Also, "human" should be plural.
* "algorithms.The" -- Missing a space, I think.
* "PEARL leverages … latent variable Z" -- This sentence doesn't make sense as written.
* "z's" -- This should not be a possessive.
* "Good sample efficiency enables fast adaptation … and performs structured exploration …" -- Isn't the first part true by definition? Why does good sample efficiency perform structured exploration?
* "a goal is a 3-d vector" -- If the goal output by the high-level policy is the XYZ coordinates of the
* "SAC, such non-hierarchical" -- Grammar doesn't make sense here.
* "such non-hierarchical RL method has been proved to perform badly before on …" -- Can you add a citation? Generally, "proved" is reserved for mathematical proofs.
* "In this paper, We have" -- "We" should not be capitalized.
------------------------ UPDATE AFTER AUTHOR RESPONSE ------------------
I thank the authors for at least reading the reviews. My concerns with experiments and clarify remain unaddressed, and are amplified by reading the other reviews. I therefore vote to "reject" this paper.


In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations. We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views. Formally, multiple views of the same sentence are mapped to close representations. On the contrary, views from other sentences are mapped further. By contrasting different linguistic views, we aim at building embeddings which better capture semantic and are less sensitive to the sentence outward form.

Overview: This work proposes to learn sentence embeddings using both contrastive learning and multiple "views" of sentences.  This work largely builds off of [1], including using the same objective, but uses a multi-view approach to modeling.
    - They apply the concept of multi-view models, specifically combining tree and linear LSTMs to learning sentence representations.
    - They prepare a new, large-scale book dataset, which is useful because the previously commonly used book dataset was taken down for legal reason.
    - They provide a fairly broad set of analyses on their model, both quantitative and qualitative, performance-driven and analysis-driven.
- Review: The ideas and models presented in this paper are not new, while the supporting experiments are not very well done or convincing. Overall, I recommend rejecting this work.
- The models are contrastively learned in that they are trained to embed "similar" sentences nearby in the embedding space, and "dissimilar" sentence far away, where "similar" sentences are defined as consecutive sentences. This method of learning textual representations is well-established in the NLP literature, mostly prominently in recent years with word embedding models like Skip-Gram and in sentence embedding models like in [1], [2], [3] (the next sentence prediction task), and several more.
- In practice, the multiple views of each sentence that this paper considers boils down to encoding the sentence with a bidirectional LSTM and a TreeLSTM and concatenating the representations from each encoder. This idea again has been established in the literature ([4], [5], [6]).
- The experiments don't seem setup to demonstrate that the multiple views are beneficial over a single view. In Table 1, there are rows for just an LSTM or just a TreeLSTM, but they seem to be trained with labeled data whereas the proposed method is trained self-supervised. A more informative comparison to demonstrate the value of using multiple views would be to train the LSTM and TreeLSTM with the same objective (and ideally model size). Overall, I don't think the claims in the paper are well-supported by the model proposed or the experiments.
- I have a number of concerns about the experiments.
    - "Models are trained on a single epoch on the entire corpus without any train-test split": so there is no early stopping? Why stop training after one epoch? Was there any indication you were overfitting the data?
    - "The training phase was stopped after 33 hours of training": Why stop there? Computational constraints? Later comments suggest this is quite premature ("training phase was completed on only 4.6M sentences among the 78M available").
    - The results seem to indicate that this method underperforming recent work significantly.
Areas of improvement
- Some of the language in the introduction and conclusion are a bit of a stretch. Using a linear and tree LSTM (based on dependency parses) doesn't really represent a "diversity of linguistic structures".
- Related work: There's no mention of pretained language models, which could be seen as a form of representation learning for language, and have been hugely impactful in NLP.
- Method
    - Missing negative in the log likelihood
    - Why do you use inner product if other works "report excellent results" with other scoring functions?
    - "assumes the underlying structure of the sentence to be a sequence, while allowing for long term dependencies": If anything, the treeLSTM more easily allows for long-term dependencies than the linear LSTM.
    - "Negative examples are obtained using the dependency Tree LSTM": I'm not totally sure how the negatives are obtained here.
    - "The target sequence is encoded using the sequential Tree LSTM, while the positive and negative samples are encoded using the ChildSum Tree LSTM": why are the sentences not all encoded with the same encoder?
    - It looks really odd that most of Table 1 is empty. Given your model, I imagine it can't have been that difficult to evaluate more baselines (BiLSTM and TreeLSTM) on the rest of the tasks.
    - It'd be nice if you could clearly indicate in Table 1 which method is yours.
- Results and Analysis
    - The standard evaluation setting for sentence embeddings would be GLUE or SuperGLUE.
    - A glaringly missing baseline is BERT (or any of its relatives), which is also self-supervised.
    - The results are underwhelming, and as the author admits, somewhat premature as training didn't seem to finish.
    - 5.2: what are the contrastive LSTM and Tree LSTM? Are those the learned encoders from the "Contrastive Tree" in Table 1, or are they trained from scratch?
    - I don't think the analyses in Sections 5.2 and 5.3 or Figure 2 are particularly useful.
- There are a noticeable number of typos. For example, in the abstract: "this linguist[ic] diversity" and "better capture semantic[s]". It'd be worthwhile to look over the paper closely for typos.
[1] AN EFFICIENT FRAMEWORK FOR LEARNING SENTENCE REPRESENTATIONS. Lajanugen Logeswaran and Honglak Lee
[2] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. Yacine Jernite, Samuel R. Bowman, David Sontag
[3] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
[4] Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang.
[5] Enhanced LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen.
[6] Improving Sentence Representations with Consensus Maximisation. Shuai Tang, Virginia R. de Sa.


Zeroth-order optimization is the process of minimizing an objective f (x), given oracle access to evaluations at adaptively chosen inputs x. In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable. We analyze our algorithm from a novel geometric perspective and present a novel analysis that shows convergence within an -ball of the optimum in O(kQ log(n) log(R/ )) evaluations, for any monotone transform of a smooth and strongly convex objective with latent dimension k < n, where the input dimension is n, R is the diameter of the input space and Q is the condition number. Our rates are the first of its kind to be both 1) poly-logarithmically dependent on dimensionality and 2) invariant under monotone transformations. We further leverage our geometric perspective to show that our analysis is optimal. Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on BBOB and MuJoCo benchmarks.

Update after rebuttal: I found the rebuttal convincing and I liked the fact that concerns regarding empirical justification were addressed. Consequently, I increase my score from "Weak Reject" to "Weak Accept".
--------------------------
This paper focuses on derivative-free, or zero-th order, optimization. That is the setting where a function may be continuous and/or smooth and/or (quasi)-convex, however, we do not have access to the gradients. As such, it is not possible to apply standard gradient descent methods. The paper proposes an algorithm for "gradientless" descend. The basic intuition is that by careful randomly sampling it is quite probable that a lower objective value will be attained. Doing so recursively can then lead to the optimum with high probability. Clearly, in such a setting it is quite important to clarify what is "careful random sampling". To this end the paper casts this as sampling from a Gaussian ball of specific radius, chosen such that the samples are with high probality below the current level set (the hyperplance of equivalent solutions f(x) as our current solution f(x_t)). The paper derives and proves various theorems on how to select the optimal radius and how to perform the sampling. Specifically, the case of strongly convex and smooth functions is analyzed, however, the paper also shows how this generalizes to functions after a monotone transformation (thus leading to quasi-convex functions) and with extra error perturbations. The proposed algorithm is compared on a synthetic experiment, and a selection of MuJoCo benchmarks.
Strengths:
+ The derivations and the theorems are non-trivial. There is some serious analysis regarding the selection of radius of Gaussian balls. I would like to congratulate the authors for this. I particularly like the extension to having a perturbing function h(x), leading to a more realistic setup.
+ I also particularly like that the algorithm is able to recover subspaces automatically. This definitely makes the algorithm much more practical and more efficient.
+ The writing and the presentation are rather clear and well taken care of. Although there are several theorems, it was not too hard to follow the flow of the paper. The algorithm boxes are also concise and clear, helping with understanding the final result.
Weaknesses:
+ Although the contributions of the work are mostly on the theoretical side, I have a hard time grasping how useful is the algorithm in practice. For one, there is the assumption of strongly convex and smooth function. Granted, there is the relaxed case of having the perturbing function h(x), howevrer, in that case it seems that the algorithm becomes a slower by an order of 60 δ k Q_g(A). How fast or slow is this in practice? Even with applying the monotone function, the algorithm becomes more practical by being applicable to quasi-convex setups. However, how realistic is that a function will in practice be strictly monotone? While most of this may be hard to be theoretically proven, they can be experimentally tested.
+ Also, given that the paper is interested in black box functions, we cannot have much information regarding the function. So, what happens when the function is not strongly convex or not always smooth? Furthermore, how realistic is to know the condition number, that is the maximum derivative (or an upper bound of it) since we do not have access to the gradients in the first place?
+ I would say that the paper could benefit from a more extensive experimental section. Currently only a single synthetic function is analyzed, also under a single monotone exponential transformation. From a more practical point of view, MuJoCo environments are also examined. However, there exist no comparisons with other methods in the literature, including ARS. Another relevant algorithm to compare with would be the stochastic tree points (Bergou et al., 2019), if not experimentally at least theoretically. In the end, it quite unclear whether the algorithm work well in practice. Some experiments that could shed light would relate to how sensitive the algorithm is to the convexity/smoothness assumptions, how sensitive the algorithm is to the perturbing function h(x), how sensitive is the algorithm to the present of a lower-dimensional subspace that needs to be discovered. And for the MuJoCo experiments, the algorithm can compare at least with ARS.
+ In the experiments it seems the paper is particularly good in high dimensions. Can this be more precisely connected to the derived theory in the discussion of the experiments? Does this relate to the better subspaces k that are discovered automatically by the algorithm?
+ It is unclear how many evaluations are needed per step, that is what is the K value in the algorithm box? Also, there are at least two balls to sample from, so twice as many evaluations, correct?
+ It is unclear why Bayesian Optimization is not considered for at least comparing experimentally. Currently, the paper discards them on the grounds that they do not provide strong theoretically guarantees. However, it would be interesting to examine at least in practice how good/bad are these algorithms in comparison to the proposed one. Two recent bayesian optimization papers that can be considered for continuous and discrete inputs are
[1] BOCK: Bayesian Optimization with Cylindrical Kernels, C. Oh, E. Gavves, M. Welling, ICML 2018
[2] BOCS: Bayesian Optimization of Combinatorial Structures, R. Baptista, M. Poloczek, ICML 2018
+ The paper does not have a conclusion. That shows great sloppiness. Also, i/n the abstract, do you mean to say k>=n or k<=n?
To conclude, I recommend weak rejection only because I am not completely convinced by the experiments and do not know if the proposed algorithm is competitive against reasonable baselines and in more complex setups. I am more than happy to upgrade my score if experiments become more clear.


Most modern learning problems are highly overparameterized, meaning that the model has many more parameters than the number of training data points, and as a result, the training loss may have infinitely many global minima (in fact, a manifold of parameter vectors that perfectly interpolates the training data). Therefore, it is important to understand which interpolating solutions we converge to, how they depend on the initialization point and the learning algorithm, and whether they lead to different generalization performances. In this paper, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which the popular stochastic gradient descent (SGD) is a special case. Recently it has been shown that, for overparameterized linear models, SMD converges to the global minimum that is "closest" (in terms of the Bregman divergence of the mirror used) to the initialization point, a phenomenon referred to as implicit regularization. Our contributions in this paper are both theoretical and experimental. On the theory side, we show that in the overparameterized nonlinear setting, if the initialization is close enough to the manifold of global optima, SMD with sufficiently small step size converges to a global minimum that is approximately the closest global minimum in Bregman divergence, thus attaining approximate implicit regularization. For highly overparametrized models, this closeness comes for free: the manifold of global optima is so high dimensional that with high probability an arbitrarily chosen initialization will be close to the manifold. On the experimental side, our extensive experiments on the MNIST and CIFAR-10 datasets, using various initializations, various mirror descents, and various Bregman divergences, consistently confirms that this phenomenon indeed happens in deep learning: SMD converges to the closest global optimum to the initialization point in the Bregman divergence of the mirror used. Our experiments further indicate that there is a clear difference in the generalization performance of the solutions obtained from different SMD algorithms. Experimenting on the CIFAR-10 dataset with different regularizers, 1 to encourage sparsity, 2 (yielding SGD) to encourage small Euclidean norm, and 10 to discourage large components in the parameter vector, consistently and definitively shows that, for small initialization vectors, 10 -SMD has better generalization performance than SGD, which in turn has better generalization performance than 1 -SMD. This surprising, and perhaps counter-intuitive, result strongly suggests the importance of a comprehensive study of the role of regularization, and the choice of the best regularizer, to improve the generalization performance of deep networks.

This paper studies the performance of the mirror gradient method when applied to the overparameterized network. The authors claim that the SMD method could find the regularized global minimize for different potential functions, in terms of minimal Bregman distance. Further experiments are carried out to back up the authors' claim.
However, the drawbacks of this paper are listed in these several points:
1) The regularization w.r.t. Bregman distance is quite different from the regularization used in network training: instead of $\|w - w_0\|$, we use $\|w\|$ more often, therefore, the virtue of sparsity shared by 1-norm is not exploited.
2) The authors' assumption in Assumption 3.1 is too fuzzy: instead of detailed analysis in the papers related to the overparameterized network, the author does not give ANY relationship between the $\epsilon$ and these three important parameters: a) network width, b) probability introduced by random initialization, c) the number of input data.
3) Therefore, according to the too strong and fuzzy assumption mentioned in the last point, Assumption 3.1 simply makes the network equals to the linear model, which leads to minor contributions according to the study of the overparameterized network.
4) The `over parameterized network' discussed now, including the work of (Li & Liang, 2018; Du et al., 2018; Azizan & Hassibi, 2019; Allen-Zhu et al., 2019; Cao & Gu, 2019), are mainly focused on the `network of infinite width', however, in the authors' experiment, the network architecture, e.g. ResNet18, is more to be a `very deep network' rather than a `very wide network'. Therefore, the authors' theory is built on Assumption 3.1, which is based on `network of infinite width', while the experiment is built on the `very deep network'. The result of the experiments is not enough to support the authors' theorem.
In conclusion, I am convinced that the authors' work is over-claimed in this paper.


End-to-end models have achieved considerable success in task-oriented dialogue area, but suffer from the challenges of (a) poor semantic control, and (b) little interaction with auxiliary information. In this paper, we propose a novel yet simple end-to-end model for response generation via mixed templates, which can address above challenges. In our model, we retrieval candidate responses which contain abundant syntactic and sequence information by dialogue semantic information related to dialogue history. Then, we exploit candidate response attention to get templates which should be mentioned in response. Our model can integrate multi template information to guide the decoder module how to generate response better. We show that our proposed model learns useful templates information, which improves the performance of "how to say" and "what to say" in response generation. Experiments on the large-scale Multiwoz dataset demonstrate the effectiveness of our proposed model, which attain the state-of-the-art performance.

This paper describes a method to incorporate candidate templates to aid in response generation within an end-to-end dialog system. While the motivation and task setup is interesting, the paper is clearly unfinished. Most jarringly, Table 2 which should contain the main results comparing the proposed RGTI model to existing baseline models is not filled in, and there appears to be no table showing results of the ablation study briefly described in Section 3.4.
Other issues include the lack of detail around any of the subtasks mention. For example, in Sec 2.1, how is semantic information used as search criteria to retrieve responses? We also need to see more details about where responses are retrieved from. In Sec 2.2, how is the splitting done to construct tuples from the candidate templates (the word "in" is dropped in the example given, and it's not clear if this is a typo or whether some processing is done over the phrases before tuple construction is done).
Other comments:
- The paper needs to be heavily proofread. There are many spelling/grammar mistakes, for example:
	- dilogue -> dialogue (Figure 2, page 2)
	- acurrate -> accurate (top of page 3)
        - "First we should retrieval responses" -> "retrieve responses" (page 3)
        - BlEU -> BLEU (page 5)


We provide a critical view of the univariate Structural Causal Model. We show that by comparing the individual complexities of univariate cause and effect, in the Structural Causal Model, one can identify the cause and the effect, without considering their interaction at all. The entropy of each variable is ineffective in measuring the complexity, and We propose We suggest to capture it the complexity by the reconstruction error of an autoencoder that operates on the list of sorted samples the percentiles of the distribution. Comparing the reconstruction errors of the two autoencoders, one for each variable, is shown to perform well on the accepted benchmarks of the field. Hence, the decision as to which of the two is the cause and which is the effect may not be based on causality but on complexity. In the multivariate case, where one can ensure that the complexities of the cause and effect are balanced, we propose a new adversarial training method that mimics the disentangled structure of the causal model. We extend the results of Zhang & Hyvrinen (2010) to the multidimensional case, showing that such modeling is only likely in the direction of causality. Furthermore, the learned model is shown theoretically to perform the separation to the causal component and to the residual (noise) component. Our multidimensional method obtains a significantly higher accuracy than the literature methods.

Edit after author rebuttal and author additions:
I have updated my score from a weak reject (3) to a weak accept (6).
Justification:
1. The authors have pointed out that I misunderstood one of their contributions: pointing out that they are demonstrating that the univariate case is an insufficient setting to test causal discovery methods because it can be done without even looking at the conditional distributions (just the marginals). This contribution seems important to orient future work. They have made this more clear in their recent upload and would likely make it even more clear in a camera-ready version.
2. The authors have made their contribution to the multivariate setting more substantial by adding evaluation on the MOUS-MEG real-world dataset and have better positioned their work relative to others by adding comparisions to multivariate extensions of PNL and CGNN.
====================================================================================================
Original Review:
Summary: The authors focus on the problem of inferring whether the causal structure X —> Y or Y —> X. They first consider the case where both X and Y are scalar (univariate) random variables and then consider the case where X and Y are vector-valued (multivariate) random variables. In the scalar case, motivated by the idea that the effect could be less entropic than the cause (due to data processing inequality), they introduce a method based on comparing reconstruction losses of X and Y and show competitive results in Tables 1 and 2. They establish that this method is not sufficient for the multivariate case in Lemma 2 and move to a new method for the multivariate case. They prove identifiability for this new method in for the multivariate case in Section 4.2 and claim state-of-the-art (SOTA) results in Table 3.

Main contributions:
- Presents a causal discovery technique for the univariate cases that only examines the marginal distributions of X and Y and seems fairly competitive (Tables 1 and 2)
- Extends the post-nonlinear identifiability analysis of Zhang & Hyv¨arinen (2009) from scalars to vectors and proved that their method will actually identify the correct causal direction
- Demonstrates competitive experimental results for both their univariate method
- Claims SOTA results for their multivariate method
Decision: I lean toward rejecting this paper because 1) I have several questions about the univariate case (see below) that would need to be resolved before I lean toward accept, 2) although I am not too familiar with the literature, I believe that this paper may be missing key related work that also uses independence testing for causal discovery (see, e.g., Heinze-Deml et al. (2017)’s Invariant Causal Prediction for Nonlinear Models), and 3) I am not yet convinced that the comparison done in Table 3 is fair and exhaustive.

Sufficient reason to accept: If the theorems in Section 4.2 are found to checkout, and the SOTA results in Table 3 are found to be fair, exhaustive comparisons to the previous SOTA, their contribution to the multivariate case would seem to be sufficient for acceptance. I believe more discussion between the authors and reviewers is necessary here.

Questions about univariate case:

1. The motivation for the first method (entropy decreasing along a Markov chain due the data processing inequality) seems to only be valid when Y := f(X), but not necessarily when Y := f(X) + E. For example, let f be the identity function and E be independent to X. How did you resolve this argument against the intuition?

2. Also, I thought the data processing inequality relates mutual information between variables, not necessarily their entropies. Can you make this connection more clear?

Context for questions 3 and 4: In Section 3.1, you write, “estimating the entropy of each random variable from its samples does not present a consistent difference between the entropies h(X)  and h(Y). Our method, therefore, computes an alternative complexity score for X and, independently, for Y.” You then go on to link the entropy to the reconstruction error (your method) in Lemma 1 and show competitive results in Tables 1 and 2.
3. Why do you want to link the reconstruction error to entropy if you found a purely entropy-based method did not work?
4. Why did the purely entropy-based method not work while your method worked if the two are linked?
Questions about multivariate case:
5. Are you certain that BivariateFit and ANM are the only models that you should be comparing against for this multivariate setting?
6. What is CGNN’s runtime? Would you be able to compare against CGNN in time for a potential camera-ready version of this paper?


We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus. This problem is often solved in two steps. The retrieval phase first reduces the solution space, returning a subset of candidate documents. The scoring phase then scores and re-ranks the documents. The algorithm used in the retrieval phase is critical. On the one hand, it needs to have high recall -otherwise some relevant documents won't even be considered in the scoring phase. On the other hand, it needs to be highly efficient, returning the candidate documents in time sublinear to the total number of documents. Unlike the scoring phase which witnessed significant advances recently due to the BERT-style cross-attention models, the retrieval phase remains less well studied: most previous works rely on the classic Information Retrieval (IR) methods such as BM-25 (token matching + TF-IDF weights). In this paper, we conduct a comprehensive study on different retrieval algorithms and show that the two-tower Transformer models with properly designed pre-training tasks can largely improve over the widely used BM-25 algorithm. The pre-training tasks we studied are Inverse Cloze Task (ICT), Body First Selection (BFS), Wiki Link Prediction (WLP) and the combination of them.

This paper studies a query-related document retrieval problem using a framework which they call “two-tower retrieval method”. The task is to learn query representation and document representation in order to retrieve query-related documents by the maximum inner product. This is a realistic setting for large-scale retrieval problem since it enables document representations to be computed once regardless of the question, and obtaining query-sensitive document representations is very expensive.
Then, the paper studies three different pretraining methods for this task, ICT (previously proposed by Lee et al 2019), and BFS & WLP (proposed by this paper).
For evaluation, the paper considers the retrieval task of question answering, based on SQuAD and Natural Questions. The combination of ICT, BFS and WLP achieves remarkable improvement over the number of baselines including BM25 and other neural-based models.
The strength of this paper is that it includes comprehensive studies on the two-tower retrieval problem. In particular, they have conducted extensive ablation studies with different train/test ratios.
However, there are some notable weaknesses of this paper as follows.
First, the benchmark relies on the recall rate instead of the end task (open-domain QA). Recall rate is not a good way to evaluate the retrieval result since a system may retrieve text which contains the answer text but is not semantically related to the question. (I understand that the paper follows Admad et al (2019), but I believe this is not a published paper.) In addition, this paper did not empirically demonstrate the relatedness between the recall rate and the end performance. This makes it very hard to compare with other papers in open-domain QA, which has been extensively studied for a few recent years.
Second, despite comprehensive studies, the fact that ICT+BFS+WLP is almost the same as ICT (93.91 vs. 94.37) means that the method does not give improvement over ICT which was already proposed in the previous study.
Third, the gap between BM25 and ICT+BFS+WLP in Table 3 and 4 are very significant (e.g. 27 vs 94 on Natural Questions), but this doesn't seem to be consistent to Lee et al (2019). (There are some differences: (1) Lee et al (2019) compares BM25 vs. ICT, but according to this paper, ICT and ICT+BFS+WLP are similar. (2) Lee et al (2019) reports the end QA performance while this paper reports the recall rate, but one of the assumptions in this paper is that recall rate and the end performance is related.) What is the explanation for this discrepancy?
(I am happy to increase the rating if my concerns are resolved during rebuttals and/or the paper includes performance on the end QA performance.)
Some questions:
1) Section 4.1 says ICT is sentence-level, BFS is paragraph-level and WLP is document-level. What does it mean? I thought, according to Section 3, all methods are paragraph-level.
2) Section 4.1: it looks like Ahmad et al (2019)’s setting is actually not entirely open-domain. Their candidate sentences/paragraphs are much less than the entire Wikipedia. Did this paper also use the same set of the candidate? In that case, it should be clearly mentioned in the paper. In addition, the data statistics are different across two papers. Did Ahmad et al (2019) include only train set whereas this submission reports train+test? In case there is an official split of train/test, why were different splits used for evaluation?
3) Also regarding the split: for each split, how much was used for development? I believe data used for the development and test should be different. In fact, rather than experimenting on different ratios of train/test, is it possible to report on official test set, while splitting the train set into 90/10 for training and development? Or, split the entire data to 90/5/5 for training/development/test?
Update on Nov 15: The revised paper resolves most of my concerns, so I am updating the score from 3 to 6.


Meta-learning is a promising strategy for learning to efficiently learn within new tasks, using data gathered from a distribution of tasks. However, the meta-learning literature thus far has focused on the task segmented setting, where at train-time, offline data is assumed to be split according to the underlying task, and at test-time, the algorithms are optimized to learn in a single task. In this work, we enable the application of generic meta-learning algorithms to settings where this task segmentation is unavailable, such as continual online learning with a time-varying task. We present meta-learning via online changepoint analysis (MOCA), an approach which augments a meta-learning algorithm with a differentiable Bayesian changepoint detection scheme. The framework allows both training and testing directly on time series data without segmenting it into discrete tasks. We demonstrate the utility of this approach on a nonlinear meta-regression benchmark as well as two meta-image-classification benchmarks.

The paper proposes a solution for using meta-learning methods without the need of determining the task segmentation a priori. Authors identify that the task segmented setting is very similar to the problem of change-point detection (CPD) and particularly, they connect the generative model of the meta-learning approach to the Bayesian recursive method of Adams and MacKay 2007. The presentation of the meta-learning problem is well done and I understand its importance within the difficulties for modelling new unobserved tasks. The notation and description of the generative model included in the problem statement section is clearly understandable for any reader, this is a very positive point. They demonstrate a deep comprehension of the BOCPD model of Adams, and its extension for the meta-learning approach is original to me. Lastly, the presentation of the MOCA meta-learning algorithm is useful for reproducibility and I see it completely applicable to other scenarios. There is a noticeable effort of describing the solution for both regression and classification problems and the empirical results conclude a positive performance of MOCA.
Overall, I consider that the paper is well-written with thorough explanations. Of course, there are some details that could be improved (I will comment this later). If done, I would be willing to increase my score. The contribution of the paper is significant to meta-learning models and I think it will help to spread the Adams’ model to other type of problems.
There is an important contribution in the paper that I have to mention and it is also relevant for the future application of the Adams model. If one reads the original BOCPD paper, in particular Eq (1), where the predictive posterior p(x_t+1|x_{1:t}) is defined from a marginalisation over the run length values, it is noticeable that this equation is not used in the final recursion of the CPD method. This is because the posterior p(r_t|x_{1:t}) is sufficient for determining if there is a CP on the t time-step or not. So the predictive posterior is never used in practice (in the original paper). When I first read Adams 2007, this detail was clear to me. Surprisingly, I find that the authors have find a practical use of this equation (Eq. (7)) and it is in the main core of the meta-learning algorithm, this is fantastic.
The details I think should be improved are:
- The conditional run-length prior p(r_t|r_{t-1}) barely appears and without any detailed description is not obvious for non-familiar readers with the Bayesian CPD approach.
- The first time I read the manuscript, the use of z_t in the BOCPD presentation made me feel a bit lost. Why not use x or y? At least specify that is a toy variable for the explanation. Later on, authors change again to the x,y notation.
- Equations 3 and 4 are too similar, this looks a bit repetitive. Why not reusing one of them or say the change from one term to another?
- If reading the literature of the BOCPD and posterior extensions, the likelihood model of the detector is often referred as the underlying predictive model (UPM). Using a similar term would help for orienting familiar readers into the solution.
- As authors should have noted on their experiments, as one makes the run-length higher, the number of parameters \eta[r_t] increases. A clear state on how this is solved would help.
- In the PCOC subsection, I find the definition of y \sim q(y) a bit weird. Using Cat(_) or Multinomial likelihood notation would be a bit better.


A significant issue in training deep neural networks to solve supervised learning tasks is the need for large numbers of labeled datapoints. The goal of semisupervised learning is to leverage ubiquitous unlabeled data, together with small quantities of labeled data, to achieve high task performance. Though substantial recent progress has been made in developing semi-supervised algorithms that are effective for comparatively small datasets, many of these techniques do not scale readily to the large (unlabeled) datasets characteristic of real-world applications. In this paper we introduce a novel approach to scalable semi-supervised learning, called Local Label Propagation (LLP). Extending ideas from recent work on unsupervised embedding learning, LLP first embeds datapoints, labeled and otherwise, in a common latent space using a deep neural network. It then propagates pseudolabels from known to unknown datapoints in a manner that depends on the local geometry of the embedding, taking into account both inter-point distance and local data density as a weighting on propagation likelihood. The parameters of the deep embedding are then trained to simultaneously maximize pseudolabel categorization performance as well as a metric of the clustering of datapoints within each psuedo-label group, iteratively alternating stages of network training and label propagation. We illustrate the utility of the LLP method on the ImageNet dataset, achieving results that outperform previous state-of-the-art scalable semi-supervised learning algorithms by large margins, consistently across a wide variety of training regimes. We also show that the feature representation learned with LLP transfers well to scene recognition in the Places 205 dataset.

The authors propose a local label propagation approach for large-scale semi-supervised learning. The approach learns a representation that tries to minimize a combination of the cross-entropy loss on the labeled data and a negative inner-product-based likelihood between the propagated pseudo-label and other examples with the same true label. The pseudo-labels on the unlabeled data are then calculated with a weighted k-NN scheme, where the weights take a heuristic correction of a soft similarity. Some further computational speedup is done with a memory cache described in an earlier work (Wu 2018b). Experimental results seem significantly superior to the competitors. The design choices are mostly justified with ablation studies.
The whole idea is interesting and the results are promising. From the current manuscript, my remaining concerns are
(1) How much contribution has readily been done by (Wu 2018a, b), and how much is the original design of the authors? From Section 3 (and without reading (Wu 2018a, b)), I cannot find a clear answer to this question. Currently it appears that the additional contribution over Wu's works is marginal.
(2) It is not clear to me how the proposed approach reaches the asserted efficiency over global label propagation approaches. In particular, each P(v_i)*Z in Equation (2) is O(N) to compute. Each w_j(v) in Equation (5) is O(K) to compute after getting all P(v_i)*Z, and then there are N (or at least N-M) such w_j(v) needed. So the total complexity is naively O(N (N-M) K). Even ignoring the K as a small constant, I cannot see how LLP is O(NM). Some running time profiling of LLP versus global LP might be helpful.
(3) For label propagation methods, it is important to understand whether the pseudo-labels are accurate and/or whether the methods might be mis-guided by the pseudo-labels. Is there any evidence on whether the pseudo-labels are accurate (absolutely, or with respect to the confidence)?
(4) For hyper-parameter selection, there is a "Learning rate is initialized to 0.03 and then dropped by a factor of 10 whenever validation performance saturates." But it is not clear how the validation set is formed, and what performance is measured. Is it a performance based on a labeled validation set (and if so, how large is the set) or unlabeled one?
I read the rebuttal. While it does not change my assessment, I thank the authors for clarifying some issues.


Convolutional Sparse Coding (CSC) has been attracting more and more attention in recent years, for making full use of image global correlation to improve performance on various computer vision applications. However, very few studies focus on solving CSC based image Super-Resolution (SR) problem. As a consequence, there is no significant progress in this area over a period of time. In this paper, we exploit the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and it can be implemented using CNN architectures. Then we develop a novel CSC based SR framework analogy to the traditional SC based SR methods. Two models inspired by this framework are proposed for pre-/post-upsampling SR, respectively. Compared with recent state-of-the-art SR methods, both of our proposed models show superior performance in terms of both quantitative and qualitative measurements.

This work exploits the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and state-of-the-art performance is achieved on popular benchmarks.
[Strengths]
- This paper is well-written and easy to follow.
[Weaknesses]
- My main concern on this work is its novelty seems to be limited. In Introduction, the authors raised 4 kinds of issues, including framework/optimization/memory/multi-scale. However, optimization and memory issues are mitigated by the CNN architectures; multi-scale issue is addressed with the help of work from Kim et al.. It seems the proposed questions are mostly addressed by the previous methods.
- It seems CISTA may be the main novelty of this work. However, no comparasions with previous ISTA methods (e.g., [R1-3]) are conducted and discussed.
- From Tabs. 2&3, the improvement is very limited. Besides, time complexity needs to be compared with the competitors, including flops or inference times.
[R1] A Fast Proximal Method for Convolutional Sparse Coding, IJCNN 2013.
[R2] Convolutional Neural Networks Analyzed via Convolutional Sparse Coding, JMLR 2017.
[R3] On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks, TPAMI 2018.


In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories. While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes. This makes deep neural nets ill-suited to continual learning. In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced. We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.

This paper considers the use of a metric learning approach in a continual/lifelong classification settings. Experiments show in the case of two tasks forgetting can be minimized by using the approach.
Methods
The proposed method appears to be a standard triplet loss. The authors add a second term to the triplet loss that is essentially making the loss a combination of the triplet and siamese loss. It’s not really explained anywhere why they do this and whether its essential to the performance.
Is there anything specific to continual learning done or is the paper essentially pointing out this existing method (metric learning + nearest neighbor) is surprisingly effective for forgetting. If this is the case the authors should present it in this way I think.
Although triplet loss can often yield reasonably performance on classification problems it tends to not perform as well as cross entropy loss, this is observed in other works as well as this one.
A major question of mine: it is not clear from the method nor experiments what samples are stored after task A for the kNN classifier. Is it all of the data samples from the previous task?
Experiments
The experimental results consider a custom continual learning setup where there is two sets of categories. Overall the experiments seem lacking at the moment in rigorous comparisons.
MNIST experimental comparisons are currently suspect. It is  very surprising that LwF does so poorly, do the authors have some explanation for this. LwF is typically a reasonable baseline for these 2 task settings (e.g. https://arxiv.org/pdf/1704.01920.pdf).  Similarly the well known EWC is shown to simply not work at all for the very task it was designed for on the MNIST dataset. LwF and EWC simply not working to any degree seem to me like  rather dramatic claims to make without any explanation.
Cryptically the fine-tuning baseline described in 4.2 is not shown here for MNIST? This seems a major oversight
CIFAR10/Imagenet Experiments
It is not clear if the baseline finetuning is done on only the top weights or the entire network. Both of these baselines should be considered. Another good baseline to consider is finetuning with cosine distance and only the top weights as in https://arxiv.org/pdf/1804.09458.pdf and other recent works should also be considered
Why do the authors not include any of the baselines from MNIST experiments here, for example LwF.
Ablations study the need for normalization and dynamic margin, it seems these are helpful for accuracy and forward transfer (and not as critical for minimizing forgetting).
The author state their method is agnostic to the task boundaries, its a bit unclear what this means in this context. The procedure is not online and the labels of the samples are being used? If the authors are referring to the need to add additional outputs to the “vanilla” model this seems like it can be trivially addressed by simply saying outputs are added the first time a new class is seen thereby making it agnostic to the boundary in the same sense as this method.
Clarity
Can be problematic at times. Although all the elements of the approach are outlined the motivations are overly wordy and repetitive making them actually hard to follow.
-(minor) first/2nd paragraph of 3.1 seems a bit redundant making it hard to follow
Overall I think the idea to consider metric learning and local adaptation for continual learning is interesting, however the current work is currently lacking in both experimental evidence (appropriate comparisons) and clear motivation/difference to existing work  for its particular instantiation of this idea.
++++Post Rebuttal++++
Thank you for your detailed responses.
The clarification about “task-agnostic” for the experiments does make them look more relevant than I had previously assessed. I do want to note that the language used for this is inconsistent with the ones used in other papers, which typically calls this a “shared-head” setting (https://arxiv.org/pdf/1801.10112.pdf, https://arxiv.org/pdf/1805.09733.pdf, https://arxiv.org/pdf/1903.08671.pdf ). It is also somewhat inconsistent with the authors own definition of “task agnostic learning” given in the introduction of this paper which implies it is something related to task boundaries at training time, in fact this is something related to availability of the task id at test time. I suggest the authors to make this more clear. Furthermore, the authors should highlight all this in the experiment text, e.g. noting EWC does poorly but this is because we use a different protocol than this and this paper etc.
Regarding the experiments under this light they do look more reasonable. Indeed it has been observed that EWC works poorly in the shared-head setting https://arxiv.org/pdf/1801.10112.pdf
Regarding the new 5 task CIFAR-10 the results are interesting, however I will point the authors to the work above (Rwalk) which also reports results in this setting better than theirs (but not by too much).
I do however still have issues regarding the memory usage of the method, specifically which data needs to be stored from previous tasks. It is still not completely clear and I find obfuscated since just one sentence not even fully answering the concern about this was added to the manuscript despite myself and another reviewer asking about it. My understanding based on the (somewhat conflicted responses) of the authors is they store a substantial amount of prior task data, but most of this is only used  at test time. For example for imagenet as much as 1000 images/class are stored for testing time. This begs the question why not use this data for training as well if it is allowed to be used by the model at testing time (and therefore preserved from the first task), why is the storage cost of this data not considered and how do the authors justify this still being a lifelong learning setup. As an alternative, why can't one use a much bigger fully parametric model that uses the same amount of storage as the authors model + stored images. It seems it is not fair to compare these to methods that cant utilize this large storage amount.
Finally its not clear if this data is stored as raw images or somehow stored as embeddings. If it is stored as embeddings this would require some discussion on how the authors avoid representation drift when the next task is training. If the authors store raw images, it means at evaluation time the entire raw dataset needs to be re-encoded, therefore the model can’t perform easily anytime inference.
Unfortunately the discussion period ended but I would have liked more clarification on this, on the other hand these pieces of information should really have been in the manuscript in the first place.
Overall, my impression of the paper is improved.  But I do think it could use some further writing revisions to emphasize/clarify key points: a) the method is not new (it says e.g. in abstract “new model” which is misleading) but its application in CL is under-explored b) the experiments show poor performance on existing methods because most of those are not designed nor work well for the shared head “task agnostic” setting, while metric learning handles it gracefully.  c) be explicit about what is the memory being stored when moving onto the next task (this should be somewhere visible and explicit) and how this is justified


From a young age we learn to use grammatical principles to hierarchically combine words into sentences. Action grammars is the parallel idea, that there is an underlying set of rules (a grammar) that govern how we hierarchically combine actions to form new, more complex actions. We introduce the Action Grammar Reinforcement Learning (AG-RL) framework which leverages the concept of action grammars to consistently improve the sample efficiency of Reinforcement Learning agents. AG-RL works by using a grammar inference algorithm to infer the action grammar of an agent midway through training. The agent's action space is then augmented with macro-actions identified by the grammar. We apply this framework to Double Deep Q-Learning (AG-DDQN) and a discrete action version of Soft Actor-Critic (AG-SAC) and find that it improves performance in 8 out of 8 tested Atari games (median +31%, max +668%) and 19 out of 20 tested Atari games (median +96%, maximum +3,756%) respectively without substantive hyperparameter tuning. We also show that AG-SAC beats the model-free state-ofthe-art for sample efficiency in 17 out of the 20 tested Atari games (median +62%, maximum +13,140%), again without substantive hyperparameter tuning.

The authors propose a method for learning macro-actions in a multi-step manner, where Sequitur, a grammar calculator, is leveraged together with an entropy-minimisation based strategy to find relevant macro-actions. The authors propose a system to bootstrap the weights of these macro-actions when increasing the policy's action space, and a system to increase the amount of data (and bias it towards macro-actions) used to learn a policy for when conditioned on this increased action-space. The authors test against a subset of the Arcade Learning Environment suite.
Overall, I'm conflicted by this paper. On one hand, the framework is interesting, and their method involves the usage and exploration of quite a few nice ideas; on the other hand, (a) the quality of the scientific contribution is hard to judge considering the significant differences between the proposed baselines and and their methods, and (b) the experimental section doesn't provide a lot of qualitative analysis and signal wrt. each component.
Furthermore, I have the following issues / questions:
1. I'm not convinced that the usage of Sequitur to build the macro-actions is sufficient to declare this work novel wrt. other macro-action papers. Sequitur usage in this case seems to be particularly overkill, since ultimately all that the method seems to be doing is finding frequent sequences of actions, which can be done quite fast (at least given the amount of training steps) simply using search and pattern matching. From my point of view, there doesn't seem to be a lot in that work that exploits the fact that the macro-actions are constructed as a "grammar" (beyond, maybe, HAR)
2. The Abandon Ship heuristics is effectively a fixed termination policy, which makes the entire setup somewhat similar to options. In this case, what is traded is learning complexity for a hyperparameter and a significant restriction in how the macro-actions terminate. Did you attempt to learn this function at all? Do you have any insights / experiments that might show how the heuristics behaves with changing values of $z$? Would it be possible to plot the distribution of attempted vs executed move lengths rather than then averages (since I doubt they would be normally distributed)?
3. Given points 1 and 2, the literature review is lacking - there's a lot of prior work done on macro-actions in both RL and robotics (planning, HRI, ...) that goes well beyond the few recent papers mentioned by the authors, and I think it might be necessary to mention work on options where the termination function is structured / biased in some way.
4. I have some doubt the experimental setup for DDQN fairly gives a fair assessment of the method. When using a pretrained features, the problem becomes significantly easier, and thus AG-DDQN potentially doesn't need to deal with the problem of learning extremely bad / noisy macro-actions. I would love to see the method trained for a more reasonable amount of frames without pre-training. Also, did the 8 / 20 atari games get chosen randomly, or were they picked based on some environment features?
5. How do the Q-values for the policy evolve with training time? The proposed methods seem to somewhat imply that the action space grows unboundedly, which might seriously destroy the policy for tasks that require much longer training. Would it be possible to add a paragraph about how the policies evolve in at least some of these environments? Are macro-actions used most of the times after some full iterations? How many <learning -> action distillation> iterations are actually done in the current experiments?
At this point, I cannot recommend the acceptance of this work, however I'd be willing to reconsider my rating if the authors address the above points.


Identifiability, or recovery of the true latent representations from which the observed data originates, is a fundamental goal of representation learning. However, most deep generative models do not address the question of identifiability, and cannot recover the true latent sources that generate the observations. Recent work proposed identifiable generative modelling using variational autoencoders (iVAE) with a theory of identifiability. However, due to the intractablity of KL divergence between variational approximate posterior and the true posterior, iVAE has to maximize the evidence lower bound of the marginal likelihood, leading to suboptimal solutions in both theory and practice. In contrast, we propose an identifiable framework for estimating latent representations using a flow-based model (iFlow). Our approach directly maximizes the marginal likelihood, allowing for theoretical guarantees on identifiability, without the need for variational approximations. We derive its learning objective in analytical form, making it possible to train iFlow in an end-to-end manner. Simulations on synthetic data validate the correctness and effectiveness of our proposed method and demonstrate its practical advantages over other existing methods.

This paper is about learning an identifiable generative model, iFlow, that builds upon a recent result on nonlinear ICA. The key idea is providing side information to identify the latent representation, i.e., essentially a prior conditioned on extra information such as labels and restricting the mapping to flows for being able to compute the likelihood. As the loglikelihood of a flow model is readily available, a direct approach can be used for learning that optimizes both the prior and the observation model.
The paper is very clear and very easy to follow. The idea is quite clear, and the direct approach is really attractive. Unfortunately, the experimental section is quite limited and does not fully study representational aspects. There is only an illustrative simulation on synthetic data, that in a sense verifies the theory. I was also not able to see the additional insight that the identifiability theory in 4.2 provides additional to Khemakhem et al. (2019). Please clarify.
My main concern is that this model is actually a supervised model that learns a mapping from u to x via z. Hence, the theoretical ‘identifiability’ guarantee needs to be stated with some care as this depends on the choice of an arbitrary u. For example if we set x = u, will learning take place? Please comment.
Overall, I like the approach but I am uncertain about the level of novelty. For such a paper, one should expect a much more involved computational study. In this respect, I feel that the paper could be accepted but it certainly feels as if it needs more computational results as otherwise the original contribution would be too incremental for ICLR standards. I am giving a provisional reject in the hope that the authors will provide convincing arguments about their original contributions for clarification.


We introduce LiPopt, a polynomial optimization framework for computing increasingly tighter upper bounds on the Lipschitz constant of neural networks. The underlying optimization problems boil down to either linear (LP) or semidefinite (SDP) programming. We show how to use structural properties of the network, such as sparsity, to significantly reduce the complexity of computation. This is specially useful for convolutional as well as pruned neural networks. We conduct experiments on networks with random weights as well as networks trained on MNIST, showing that in the particular case of the ∞ -Lipschitz constant, our approach yields superior estimates as compared to other baselines available in the literature.

In this paper, the authors introduce a framework for computing upper bounds on the Lipschitz constant for neural nets. The main contribution of the paper is to leverage the sparsity properties of typical feed-forward neural nets to reduce the computational complexity of the algorithm. Through experiments, the authors show that the proposed algorithm computes tighter Lipschitz bounds compared to baselines.
The approach proposed in the paper looks interesting. Although the presentation can be made clearer in places. For example, in equation (4), it would be helpful to explicitly state over which parameters the max is taken. There's also a number of small typos that need to be fixed. For example: "We refer to d as the depth, and we we focus on the case where fd has a single real value as output." on page 1.
I found the proposed algorithm and the discussions in Section 2 and 3 interesting, although I am not familiar enough with the literature on polynomial optimization to evaluate whether there is any significantly new idea presented in these sections. I found section 4 very interesting too, and very important towards making the algorithm actually computationally tractable. I have a couple of concerns with the rest of the paper however, which `I describe below:
1. It is nice that upper bounds for the local Lipschitz constant can be incorporated easily into the formulation. I would have liked to see some experiments on evaluating local Lipschitz constants though, and how they compare with other methods, since this is a very popular setting in which such techniques are used nowadays.
2. The paper overall I think would benefit from a better experimental evaluation. It would be interesting to see how much the sparsity pattern in convnets affect results compared to other baselines. It would also be interesting to see how the bound degrades as the network grows bigger, and in particular as the depth increases.
Given the lack of thorough experiments in the paper, I am giving the paper a borderline rating. I am however willing to increase my score based on discussions with the authors and other reviewers.
===================================
Edit after rebuttal:
The latest draft addresses most of my concerns, and I am happy to recommend accepting this paper now.