People are incredibly skilled at imitating others by simply observing them. They achieve this even in the presence of significant morphological differences and capabilities. Further, people are able to do this from raw perceptions of the actions of others, without direct access to the abstracted demonstration actions and with only partial state information. People therefore solve a difficult problem of understanding the salient features of both observations of others and the relationship to their own state when learning to imitate specific tasks. We can attempt to reproduce a similar demonstration via trial and error and through this gain more understanding of the task space. To reproduce this ability an agent would need to both learn how to recognize the differences between itself and some demonstration and at the same time learn to minimize the distance between its own performance and that of the demonstration. In this paper we propose an approach using only visual information to learn a distance metric between agent behaviour and a given video demonstration. We train a Recurrent Neural Network (RNN)-based Siamese model to compute distances in space and time between motion clips while training an Rienforcement Learning (RL) policy to minimize this distance. Furthermore, we examine a particularly challenging form of this problem where the agent must learn an imitation based task given a single demonstration. We demonstrate our approach in the setting of deep learning based control for physical simulation of humanoid walking in both 2D with 10 degrees of freedom (DoF) and 3D with 38 DoF.

Brief summary:
This work proposes a way to perform imitation learning from raw videos of behaviors, without the need for any special time-alignment or actions present. They are able to do this by using a recurrent siamese network architecture to learn a distance function, which can be used to provide rewards for learning behaviors, without the need for any explicit pose estimation. They demonstrate effectiveness on 2 different locomotion domains.
Overall impression:
Overall, my impression from this paper is that the idea is to use a recurrent siamese network to learn distances which make sense in latent space and provide rewards for RL. This is able to learn interesting behaviors for 2 tasks. But I think the writing needs significant work for clarity and completeness, and there needs to be many more baseline comparisons.
Abstract comments:
trail and error -> trial and error
Introduction comments:
Alternative reasons why pose estimation won’t work is because for any manipulation tasks, you can’t just detect pose of the agent, you also have to detect pose of the objects which may be novel/different
Few use image based inputs and none consider the importance of learning a distance function in time as well as space -> missed a few citations (eg imitation from observation (Liu, Gupta, et al))
Therefore we learned an RNN-based distance function that can give reward for out of sync but similar behaviour -> could be good to emphasize difference from imitation from observation (Liu, Gupta, et al) and TCN (Semanet et al), since they both assume some sort of time alignment
Missing related work section. There is a lot of related work at this point and it is crucial to add this in. Some things that come to mind beyond those already covered are:
1. Model-based Imitation Learning from State Trajectories
2. Reward Estimation via State Prediction
3. infoGAIL
4. Imitation from observation
5. SFV: Reinforcement Learning of Physical Skills from Videos
6. Universal planning networks
7. https://arxiv.org/abs/1808.00928
8. This might also be related to VICE (Fu, Singh et al), in that they also hope to learn distances but for goal images only.
It seems like there is some discussion of this in Section 3.1, but it should be it’s own separate section.
Section 3 comments:
a new model can be learned to match this trajectory using some distance metric between the expert trajectories and trajectories produced by the policy π -> what does this mean. Can this be clarified?

The first part of Section 3 belongs in preliminaries. It is not a part of the approach.
Section 3.2
Equations 9 and 10 are a bit unnecessary, take away from the main point
What does distance from desired behaviour mean? This is not common terminology and should be clarified explicitly.
Equation 11 is very confusing. The loss function is double defined.  what exactly Is the margin \rho (is it learned?) The exact rationale behind this objective, the relationship to standard siamese networks/triplet losses like TCN should be discussed carefully. This is potentially the most important part of the paper, it should be discussed in detail.Also is there a typo, should it be || f(si) - f(sn)|| if we want it to be distances? Also the role of trajectories is completely not discussed in equation 11.
Section 3.3
The recurrent siamese architecture makes sense, but what the positive and negative examples are, what exactly the loss function is, needs to be defined clearly. Also if there are multiple demonstrations of a task, which distance do we use then?
The RL simulation environment is it made in-house, based on bullet or something else?
Data augmentation - how necessary is this for method success? Can an ablation be done to show the necessity of this?
Algorithm 1 has some typos
- > is missing in line 3
- Describe where reward r is coming from in line 10
Section 4.1
Walking gate -> walking gait
There are no comparisons with any of the prior methods for performing this kind of thing. For example, using the pose estimation baseline etc. Using the non-recurrent version. Using TCN type of things. It’s not hard to run these and might help a lot, because right now there are no baseline comparisons


Link prediction in simple graphs is a fundamental problem in which new links between nodes are predicted based on the observed structure of the graph. However, in many real-world applications, there is a need to model relationships among nodes which go beyond pairwise associations. For example, in a chemical reaction, relationship among the reactants and products is inherently higher-order. Additionally, there is need to represent the direction from reactants to products. Hypergraphs provide a natural way to represent such complex higher-order relationships. Even though Graph Convolutional Networks (GCN) have recently emerged as a powerful deep learning-based approach for link prediction over simple graphs, their suitability for link prediction in hypergraphs is unexplored -we fill this gap in this paper and propose Neural Hyperlink Predictor (NHP). NHP adapts GCNs for link prediction in hypergraphs. We propose two variants of NHP -NHP-U and NHP-D -for link prediction over undirected and directed hypergraphs, respectively. To the best of our knowledge, NHP-D is the first method for link prediction over directed hypergraphs. Through extensive experiments on multiple real-world datasets, we show NHP's effectiveness.

[Relevance] Is this paper relevant to the ICLR audience? yes
[Significance] Are the results significant? somewhat
[Novelty] Are the problems or approaches novel? rather incremental
[Soundness] Is the paper technically sound? yes
[Evaluation] Are claims well-supported by theoretical analysis or experimental results? marginal
[Clarity] Is the paper well-organized and clearly written? okay
Confidence: 2/5
Seen submission posted elsewhere: No
Detailed comments:
In this work, the authors propose an approach to the (hyper-) link prediction problem in both directed and undirected hypergraphs. The approach first applies an existing dual transformation to the hypergraph such that the link prediction problem (in the primal) becomes a node classification problem in the dual. They then use GCNs to classify the (dual) nodes. Experimentally, the proposed approach marginally outperforms existing approaches.
=== Major comments
I found the novelty of the proposed approach rather limited. The proposed approach essentially just concatenates three existing strategies (dual reformulation from Scheinerman and Ullman, GCNs from Kipf and Welling, and negative sampling which is common in many communities, e.g., Han and Chen, but many others, as well). I believe the contribution for link prediction in directed hypergraphs is a more novel contribution, however, I had difficulty following that discussion.
It is difficult to interpret the experimental results. Tables 3 and 6 do not include a measure of variance. Thus, it is not clear if any of the results are statistically significant. It is also not clear whether the “10 trials” mentioned in the figure captions correspond to a 10-fold cross-validation scheme or something else. It is unclear to me what the random feature matrix for the metabolic network is supposed to me or do. It is also unclear to me why “fake papers” are needed for the citation networks; it is clear that “fake author lists” are needed for negative sampling, but it seems they could be attached to existing papers. Similarly, it is unclear how the set of candidate edges (\mathcal{E}) was chosen.
I appreciate that the authors made the code available. I did not run it, but I did have a look, and I believe it could be adapted by others without an unreasonable amount of work.
=== Minor comments
This work is very similar to the arXiv submission 1809.09401. To the best of my knowledge, though, that work has not yet been published in a peer-reviewed venue, so I do not consider it a problem that it is not cited here.
According to Tables 1 and 2, iAF692 and iHN637 datasets are smaller than the other datasets except DBLP; those two are also less dense than DBLP. According to Table 3, NHP-U seems noticeably better than SHC and CMM on the, while does not appear very significant in the other cases. Is there some relationship between NHP’s performance and the size/density of the graph? or is there some other explanation for this behavior?
Related to the above point, Table 3 shows that the performance on the undirected versions for those two datasets is better than on the other two metabolic networks, while Table 6 shows the opposite for the directed versions. Is there some explanation for this? For example, are there qualitative differences in the size of the hypernodes?
The described strategy for negative sampling seems as though it selects “easy” negative samples, in the sense that they are far away from observed positives; thus, they are also likely far away from any sort of decision boundary. How does the performance change if more “difficult” (or just uniformly random) negative samples are chosen?
I believe Recall@100 (or Precision@100, or @$\Delta E$, etc.) is a more meaningful value to report in Tables 4 and 7, rather than the raw number of edges. That is, it would be more helpful to report something so that numbers across datasets are at least somewhat comparable.
=== Typos, etc.
In Equation (4), the “k” index in d_{ijk} is in {1,2}, but in the text, it is in {0,1}.
“table 2” -> “Table 2”, and many other similar examples throughout the paper.
“higher-order etc.” -> “higher-order, etc.”
“GCN based” -> “GCN-based”, and similar in several places in the paper
“a incomplete” -> “an incomplete”


LEARNING FROM POSITIVE AND UNLABELED DATA WITH A SELECTION BIAS
We consider the problem of learning a binary classifier only from positive data and unlabeled data (PU learning). Recent methods of PU learning commonly assume that the labeled positive data are identically distributed as the unlabeled positive data. However, this assumption is unrealistic in many instances of PU learning because it fails to capture the existence of a selection bias in the labeling process. When the data has selection bias, it is difficult to learn the Bayes optimal classifier by conventional methods of PU learning. In this paper, we propose a method to partially identify the classifier. The proposed algorithm learns a scoring function that preserves the order induced by the class posterior under mild assumptions, which can be used as a classifier by setting an appropriate threshold. Through experiments, we show that the method outperforms previous methods for PU learning on various real-world datasets.

The paper proposes an approach to learning from positive and unlabelled data with a sample selection bias. Specifically, it is assumed that the observed positive instances are not necessarily drawn iid from the true positive distribution: rather, there is some bias as to which positive examples are selected. Under an assumption on the selection probability being proportional to the true probability, it is established that one may equally rank instances based on their probability of being labelled. Two algorithms are proposed for this task.
Learning under sample selection bias is an important and interesting problem. It is also arguably more realistic than the classic PU learning setting. The paper proposes a reasonable solution, which builds on some recent advances in the literature on PU learning.
My only critique is that the results are somewhat unsurprising in light of existing work on this topic (the idea of constructing unbiased risk estimators), and also on the topic of learning from label loans. Further, I believe some clarifications would better position the contributions of the paper, both in terms of strengths and limitations. More specifically:
- it seems the problem could be cast as a (interesting) special case of learning from instance dependent label noise. The assumption of the selection (i.e., label flip) probability preserving the ordering of the true class probability has a fair amount of precedent in these works; see, e.g.,
Bylander '97, Learning probabilistically consistent linear threshold functions
Du and Cai '15, Modelling class noise with symmetric and asymmetric distributions
Bootkrajang '16, A generalised label noise model for classification in the presence of annotation errors
It is in light of these works that I do not find Theorem 1 surprising. I note that the sample-selection bias setting could be seen as an interesting special case, but some discussion on the connection seems prudent.
- like in the above works, the proposed approach does not construct an unbiased estimator to the underlying risk. Instead, what is shown in e.g. Theorem 3 is that the Bayes-optimal solution to the risk is sensible. This is of course a minimal desiderata for any learning method, but unlike approaches for the classic PU learning setting, the lack of unbiasedness implies that minimizing over a restricted function class F may result in quite different solutions than if we had access to the true labels. Again, this isn't a limitation unique to this particular work, but I did feel the point could be made a little more explicit.
- also like the above work, there isn't a clear way of estimating P(y = 1). As this is crucial for the final risk estimate, it somewhat restricts the universality of the approach.
- with regards to the two algorithms proposed, both go about estimating the underlying "noisy" class probability (i.e., the probability of an instance being selected for labeling), just with different losses. While the logistic or "LSIF" loss are certainly valid choices, one could use any number of other similar loss (e.g., the exponential loss from class-probability estimation, or the "KLIEP" loss from density ratio estimation). Of course the specific choice of LSIF e.g. can be motivated since it has a closed-form solution, but the basic point is that the two approaches really boil down to changing the underlying loss function. This point could also be clarified.
Other comments:
- I believe the Elkan & Noto paper operates in the censoring rather than case-controlled setting.
- there are a few grammatical issues: e.g.., "Several recent researches", "is to find anomaly data"
- I don't follow how the case-controlled setting is "more general" than the censoring setting, as claimed in Sec 2; do you mean it is more practically realistic?
- it is correct to say in 2.1 that one cannot estimate p(y = 1 | x) from only PU data without assumptions. The next sentence states that a typical assumption that is thus made is SCR. However, this also does not guarantee that we can estimate the probability, since estimating p(y = 1) is also not possible without even further assumptions (see e.g. the mutually contaminated distribution work of Scott et al., 2013).
- in Defn 1, it would be clearer to explicate the dependence of all quantities on r.
- it is interesting that one achieves the BEP with the choice of threshold given by (2). But given that p(y = 1) is in general hard to estimate, it seems one could equally cast the problem of estimating p(y = 1) as the problem of choosing a good threshold? (This of course ignores the fact that we ostensibly need p(y = 1) when constructing the risk estimate.)
- restricting attention to scorers with output in [eps, 1 - eps] is a little strange. I assume this is in order to avoid solutions at +- infinity, which is a well-known problem with the logistic loss. It may be more natural to simply state that you operate with the extended real numbers.
- in the proof of Thm 3, I don't see the need to go through an infinite dimensional Lagrangian route. Since one is optimizing over all possible measurable functions, can one not (under suitable regularity conditions on the distribution & loss) simply compute the minimizer point wise for each x? This optimization would be a one-dimensional problem over predictions the domain [eps, 1 - eps]. The "inner risk" to be optimized (in the sense of Steinwart '06, "How to compare different loss functions and their risks") would I believe be a convex function, admitting exactly the minimizer claimed in the statement of the theorem.
- it is a bit confusing to move from F to \hat{F} as the function class.


This paper revisits the Random Walk model for sentence embedding in the context of non-extensive statistics. We propose a non-extensive algebra to compute the discourse vector. We argue that by doing so we are taking into account high non-linearity in the semantic space. Furthermore, we show that by considering a non-extensive algebra, the compounding effect of the vector length is mitigated. Overall, we show that the proposed model leads to good sentence embedding. We evaluate the embedding method on textual similarity tasks.

PAPER SUMMARY:
This paper introduces a non-extensive statistic random walk model to generate sentence embedding while accounting for
high non-linearity in the semantic space.
NOVELTY & SIGNIFICANCE:
I am not sure what the main focus of this paper is. It seems accounting for non-linearity in the semantic space while generating sentence embedding has already been achieved by existing LSTM models -- the goal seems to be more about interpretability and computational efficiency but the paper did not really discuss these in detail (more on this later).
In terms of the proposed solution, I am also not sure what is the significance of using non-extensive statistic in this context. In fact, the background section gave the impression that the non-linear form of q-exponential is the main reason to advocate this approach. But, if it is only about handling non-linearity, there are plenty of alternatives and it is important to point out exactly what advantages non-extensive statistic has over the existing literature (e.g., why is it more interpretable than LSTM). Please expand the respective background section to clarify this.
TECHNICAL SOUNDNESS:
There are parts of the technical exposition that appear confusing and somewhat incoherent. For instance, what exactly is this confounding effect of vector length & why do we need to address this issue if according to the Section 2.2, it has already been addressed in the same context?
Section 2.2 seems to discuss this effect but the exposition is unclear to me. The authors start with an example and a bunch of assumptions that lead to a contradiction.
It is then concluded that the cause of this is due to the linearity assumption (what about the other assumptions?) in estimating the discourse vector.
I do not really follow this reasoning and it would be good if the authors can elaborate more on this.
CLARITY:
The paper seems to focus too much on technical details and does not give enough discussion on its positioning. The significance of the proposed solution with respect to the literature remains unclear.
EMPIRICAL RESULTS:
I am not an expert in this field and cannot really judge the significance of the reported results. I do, however, have a few questions: in all benchmarks, are the algorithms tested on a different domain than the domain it was trained on?
Have the authors compared the proposed sentence embedding framework with the LSTM literature mentioned in the introduction? I noticed there was a LSTM AVG in the comparison table.
Is that the simple averaging scheme mentioned in the introduction when the authors discussed transferrable sentence embedding?
Is there any reason for not comparing with RNN (Cho et al., 2014)?
In terms of the computation processing cost, how efficient is the proposed method (as compared to existing literature)?


At internet scale, applications collect a tremendous amount of data by logging user events, analyzing text, and collecting images. This data powers a variety of machine learning models for tasks such as image classification, language modeling, content recommendation, and advertising. However, training large models over all available data can be computationally expensive, creating a bottleneck in the development of new machine learning models. In this work, we develop a novel approach to efficiently select a subset of training data to achieve faster training with no loss in model predictive performance. In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model. Extensive experiments show that our approach leads to a 1.6× and 1.8× speed-up on CIFAR10 and SVHN by selecting 60% and 50% subsets of the data, while maintaining the predictive performance of the model trained on the entire dataset.

# Summary
The paper presents a method for identifying and selecting the most informative subset of the training dataset in order to reduce training time while maintaining test accuracy. The method consists of training a proxy model that is smaller and has been trained for fewer epochs, and which can optionally be ensembled. Experiments show promising results, indicating that some datasets can be reduced to half the size without impacting model performance.
# Quality
The paper appears sound and of good quality. Background literature is cited and the proposed method is discussed in sufficient detail.
I would, however, like to see some additional comparative experiments. All experiments are constructed to show that the method can indeed achieve accuracy comparable to the full model but with a smaller training set. I would like to see how it compares to existing strategies -- are there any reason to pick this method over existing ones?
Since the last sentence in section 2 states that the proposed method is orthogonal to previous subsampling techniques, and therefore can be combined with any of them, it would be interesting to see how SVP compares to these and whether a combination of, say, SVP and importance sampling will in fact achieve better performance than the importance sampling on its own.
Additionally, given the model's high resemblance to active learning, it would be interesting to see it compared to some prominent active learning methods.
# Clarity
The paper reads quite well. I particularly like the paragraph headlines, which makes it easy to get an overview of the paper.
The figures are generally nice and readable, except for figure 3, which I don't understand. Maybe I am missing it, but I can't find an explanation for what the rows and columns indicate, and the labels themselves should also be increased in size.
# Originality
I do not find the paper particularly novel. To me, the proposed method seems to be a variant of active learning, not orthogonal to this as it is claimed in section 2. The choice of surrogate model and uncertainty metric might be new, but the method itself boils down to uncertainty sampling, a well-known strategy in active learning.
However, I am happy to change my mind if the authors can explain to me exactly how their method differs from active learning.
# Significance
While techniques for speeding up training without sacrificing performance are, of course, always interesting, I find the proposed method to be rather incremental and not significant enough for ICLR. It would be better suited as a workshop paper.
# Other notes
In the last paragraph of section 1, you write that "Our proposed framework is robust to the choice of proxy model architecture." I am not sure what you mean by this. Do you mean that one can choose any model as the proxy (which is clearly correct) or do you mean that the method is "proxy agnostic" in the sense that any proxy model will work better than no proxy? If the latter is the case, I would like some arguments for this. Also, if the method is indeed proxy agnostic, it should be possible to remove the proxy completely and select the data in some other way.


LEARNING GRID CELLS AS VECTOR REPRESENTA- TION OF SELF-POSITION COUPLED WITH MATRIX REPRESENTATION OF SELF-MOTION
This paper proposes a representational model for grid cells. In this model, the 2D self-position of the agent is represented by a high-dimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that transforms the vector. Each component of the vector is a unit or a cell. The model consists of the following three sub-models. (1) Vector-matrix multiplication. The movement from the current position to the next position is modeled by matrix-vector multiplication, i.e., the vector of the next position is obtained by multiplying the matrix of the motion to the vector of the current position. (2) Magnified local isometry. The angle between two nearby vectors equals the Euclidean distance between the two corresponding positions multiplied by a magnifying factor. (3) Global adjacency kernel. The inner product between two vectors measures the adjacency between the two corresponding positions, which is defined by a kernel function of the Euclidean distance between the two positions. Our representational model has explicit algebra and geometry. It can learn hexagon patterns of grid cells, and it is capable of error correction, path integral and path planning.

Updated score from 6 to 7 after the authors addressed my comments below.
Previous review:
This paper builds upon the recent work on computational models of grid cells that rely on trainable (parametric) models such as recurrent neural networks [Banino et al, 2018; Cueva & Wei, 2018]. It focuses entirely on path integration in 2D and 3D, from velocity inputs only, and it relies on two sub-networks: the motion model (an RNN) and the localization model (a feed-forward network). The avowed goal of the paper is to build a very simple and linear model for grid cells.
By linearly embedding the position x into a high-dimensional hidden vector v(x) (e.g., 96 elements), it can model motion using a linear model relying on matrix-vector multiplication: v(x + dx) = M(dx) v(x), where dx is the 2D or 3D displacement, v(.) is a vector and M(.) is a matrix. The embeddings v(.) are learnable and the paper assumes a square or cubic grid of N*N or N*N*N possible positions x (with N=40); these embeddings are also normalized to unit length and obey the kernel constraint that the dot-product between any two positions' vectors v(x) and v(y) is a Gaussian or an exponential function. The motion matrix is represented as block diagonal, where each block is a rotation of subvector v_k(x) into v_k(x + dx), where each block corresponds to a specific grid cell, and where the diagonal block is further expressed as a quadratic function of dx_1, dx_2, dx_3 elements of the displacement vector.
The strengths of the paper are that:
1) The supervision of the localization subnetwork only depends on Euclidean proximity between two positions x and y and therefore uses relative positions, not absolute ones. Similarly, the path integration supervision of the motion model uses only relative displacements.
2) The resulting rate maps of the hidden units seem perfect; the model exhibits multi-scale grid behaviour.
3) The idea of using disentangled blocks, rather than injecting noise or using dropout and a softmax bottleneck as in [Banino et al, 2018], is interesting.
4) The model accumulates little path integration error over 1000 step-long episodes.
The weakness of the paper is its simplicity:
1) The assumption that A(x, y) can be modeled by a Gaussian or exponential (Laplacian?) kernel is limiting, in particular for positions x and y that are far apart.
2) There is no discussion about what egocentric vs. allocentric referentials, and dx is assumed to be aligned with (x, y) axes (which are also the axes defining the bounding box of the area).
3) Unlike the other work on learning path integration using an RNN, the linear matrix model can only handle allocentric displacements dx_1, dx_2 (and optional dx_3 in 3D).
4) No consideration is given to non-square areas: would the network also exhibit grid-like behavior if the area was circular?
5) What happens if the quadratic parameterisation of block diagonals is dropped?
6) The paper did not use metrics accepted in the neuroscience community for computing a gridness score of the grid cells (although the grid cell nature is evident). There should however be metrics for quantifying how many units represent the different scales, offsets and orientations.
7) The authors did not consider (but mentioned) embedding locations from vision, and did not consider ambiguous position embeddings.
The experiments about path planning are unconvincing. First of all, the algorithm requires to input absolute positions of every obstacle into equation (9) - (10), which assumes that there is perfect information about the map. Secondly, the search algorithm is greedy and it is not obvious how it would handle a complex maze with cul-de-sac. Saying that "there is no need for reinforcement learning or sophisticated optimal control" is very misleading: the problem here is simplified to the extreme, and fully observed, and any comparison with deep RL algorithms that can handle partial observations is just out of place.
In summary, the authors have introduced an interesting and elegant model for grid cells that suffers from simplifications. The part on path planning should be cut and replaced with more analysis of the grid cells and an explanation of how the model would handle egocentric velocity.


There are myriad kinds of segmentation, and ultimately the "right" segmentation of a given scene is in the eye of the annotator. Standard approaches require large amounts of labeled data to learn just one particular kind of segmentation. As a first step towards relieving this annotation burden, we propose the problem of guided segmentation: given varying amounts of pixel-wise labels, segment unannotated pixels by propagating supervision locally (within an image) and non-locally (across images). We propose guided networks, which extract a latent task representationguidance-from variable amounts and classes (categories, instances, etc.) of pixel supervision and optimize our architecture end-to-end for fast, accurate, and dataefficient segmentation by meta-learning. To span the few-shot and many-shot learning regimes, we examine guidance from as little as one pixel per concept to as much as 1000+ images, and compare to full gradient optimization at both extremes.
To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances. Our segmentor concentrates different amounts of supervision of different types of classes into an efficient latent representation, non-locally propagates this supervision across images, and can be updated quickly and cumulatively when given more supervision.

Summary
This paper proposes to formulate diverse segmentation problems as a guided segmentation, whose task is defined by the guiding annotations.
The main idea of this paper is using meta-learning to train a single neural network performing guidance segmentation.
Specifically, they encode S annotated support image into a task representation and use it to perform binary segmentation.
By performing episodic optimisation, the model's guidance to segmentation output is defined by the task distribution.
Strength
Learning a single segmentation algorithm to solve various segmentation problem is an interesting problem that worth exploring.
This paper tackles this problem and showed results on various segmentation problems.
Weakness
The proposed method, including the architecture and training strategy, is relatively simple and very closely related to existing approach. Especially, the only differences with the referenced paper (Shaban et al., 2017) is how the support is fused and how multiple guidance could be handled, which can be done by averaging. These differences are relatively minor, so I question the novelty of this paper.
This paper performs experiments on diverse tasks but the method is compared with relatively weak baselines absolute performance looks bad compared to existing algorithms exploiting prior knowledge for each of the tasks.
For example, the oracle performance in semantic segmentation (fully supervised method) is 0.45 IOU in PASCAL VOC dataset, while many existing algorithms could achieve more than 0.8 mean IOU in this dataset.
In addition, I question whether foreground / background baseline is reasonable baseline for all these tasks, because a little domain knowledge might already give very strong result on various segmentation tasks.
For example, in terms of video segmentation, one trivial baseline might include propagating ground truth labels in the first frame with color and spatial location similarity, which might be already stronger than the foreground / background baseline.
There are some strong arguments that require further justification.
- In 4.3, authors argue that the model is trained with S=1, but could operate with different (S, P).
However, it's suspicious whether this would be really true, because it requires generalisation to out-of-distribution examples, which is very difficult machine learning problem. The performance in Figure 5 (right) might support the difficulty of this generalisation, because increasing S does not necessarily increase the performance.
- In 5.3, this paper investigated whether the model trained with instances could be used for semantic segmentation. I think performing semantic segmentation with model trained for instance segmentation in the same dataset might show reasonable performance, but this might be just because there are many images with single instance in each image and because instance annotations in this dataset are based on semantic classes. So the argument that training with instance segmentation lead to semantic segmentation should be more carefully made.
Overall comment
I believe the method proposed in this paper is rather incremental and analysis is not supporting the main arguments of this paper and strength of the proposed method.
Especially, simple performance comparison with weak baselines give no clues about the property of the method and advantage of using this method compared to other existing approaches.


LEARNING FINITE STATE REPRESENTATIONS OF RECURRENT POLICY NETWORKS
Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.

This paper proposes a method to learn a quantization of both observations and hidden states in an RNN. Its findings suggest that many problems can be reduced to relatively simple Moore Machines, even for complex environments such as Atari games.
The method works by pretraing an RNN to learn a policy (e.g. through the A3C algorithm), and then training pairs of encoder/decoder networks with a quantizing forward pass and a straight-through backpropagation. The learned quantizations can then be used to build a Moore Machine, which itself can be reduced with FSM reduction algorithms, yielding a discrete, symbolic approximation of the inner workings of RNNs, that could in principle be interpreted more easily than latent embedding spaces.
One downside of this paper is that it promises an exciting method to analyse the inner workings of RNNs, but then postpones this analysis to later work. Understandably, the synthetic experiments take some space and shows that the proposed method works as expected when the problem is amenable to discretization; maybe some parts of this could be in the appendix?
Another downside is that there is little indication of the computational implications of the method. The method was evaluated on a fairly small set of hyperparameters, and there are no indication of how long the optimization and finetuning takes. Presumably, minimizing a Moore Machine has been studied for decades, but how long does minimizing the 1000s of states in Atari games take? A second or an hour?
The paper is fairly well written and easy to understand. The method seems well grounded, although I'm not familiar enough with the quantization literature to detect if something important is missing. I think this is a great tool that hopefully will be used to try to understand the memory mechanisms of RNNs.
I think the proposed method (and the fact that it works in simple cases) warrants acceptance, but I think more experimental work would make this a great contribution. Since there is no reason for quantization to improve performance if it is done after training, then more emphasis should be put on the interpretability of the discretization; yet it is lacking in the current work. Some Atari games are known to require various amounts of memory, this could be analysed. Some other Atari games are known to be hard to solve, what happens to the RNN when the agent fails to achieve an optimal policy might also show up in the subsequent discretization and be interesting to analyse.
Comments:
- In atari, you can have access to the RAM and from it, using exactly the same mechanisms and maybe a bit of tabular MDPs, you should be able to recover the optimal MM.
- It is good that the authors report their failure to train MMNs from scratch; IMO this says something about the straight through estimators' limits. Measuring how sensible these things are to change in their target distribution and comparing to previous uses of ST in quantization works could be interesting.
- in Section 8 (appendix) "Grammer" should be "Grammar"
- All the (PO)MDPs that you analyse arguably have finite state spaces, and you set the ALE to be deterministic. What happens in continuous stochastic environments?
- Do you think a similar technique could be used to recover a (possibly stochastic) MDP instead of a Moore Machine? It would be interesting to see MDP reduction methods applied to a learned MDP.


In this study, we analyze the input-output behavior of residual networks from a dynamical system point of view by disentangling the residual dynamics from the output activities before the classification stage. For a network with simple skip connections between every successive layer, and for logistic activation function, and shared weights between layers, we show analytically that there is a cooperation and competition dynamics between residuals corresponding to each input dimension. Interpreting these kind of networks as nonlinear filters, the steady state value of the residuals in the case of attractor networks are indicative of the common features between different input dimensions that the network has observed during training, and has encoded in those components. In cases where residuals do not converge to an attractor state, their internal dynamics are separable for each input class, and the network can reliably approximate the output. We bring analytical and empirical evidence that residual networks classify inputs based on the integration of the transient dynamics of the residuals, and will show how the network responds to input perturbations. We compare the network dynamics for a ResNet and a MultiLayer Perceptron and show that the internal dynamics, and the noise evolution are fundamentally different in these networks, and ResNets are more robust to noisy inputs. Based on these findings, we also develop a new method to adjust the depth for residual networks during training. As it turns out, after pruning the depth of a ResNet using this algorithm,the network is still capable of classifying inputs with a high accuracy.

In this paper the authors analyse the role of residuals in the performance of residual networks (res-nets) with shared layer weights and propose a method to adapt the depth of such networks during training. They exploit  the fact that res-nets identical blocks / shared layer weights are discrete time dynamical systems to conduct their analysis. Their main contribution seems to be an empirical evaluation of the role of transient dynamics in the performance of such res-nets. Experiments are done on a toy dataset on concentric circles and MNIST data.
I find the scope of the paper very narrow (res-nets with shared weights) even though there seem to be quite some other papers addressing this problem.
Clarity and quality of writing. It seems to me that the paper could  be much better written, the authors present long trains of thoughts and sometimes unsubstantiated arguments and at times there seems to be no clear storyline. In particular I would strongly  suggest a rewrite of  Sections 1, 2.1, and 4.  The storyline often very sketchy and is littered with hypothetical claims or less relevant information that can distracts and confuse the reader. I find that the hypotheses are not well formulated,   the claimed contributions of the paper don't seem to be significant enough and they should also be better connected to the experimental sections. Sometimes there are some odd choices of concepts/words such as "softmax algorithm" and "classification behaviour."
Technical quality and contribution. The main technical contribution of the paper seems to be the observation that  that the solution of an  ODE if the integral of the RHS an that one can derive an ODE for the residuals only. I don't find  these results significant/relevant enough to justify the publication of this paper.  I expected a better written and more informative Section 2.1: some approximations of rates of convergence, a bit more about basins of attraction. I am also a bit skeptical about the claim that the analysed network us a prey-predator model.
Experimental section. I find the description of experiments in Sec 4 very hard to read. The metrics are not clearly defined (not sure visualisations serve the purpose either), and the performed experiments are not well motivated and explained, for example in Section 4/P2 while I think I understand what the authors want to show (path of relevant neurons), I find the purpose of the whole experiment not very relevant.  I better analysis of the number of fixed points for classification tasks should be added, comparison of resulting features to other methods such as simple MLPs with same last layer could help . More relevant datasets should be be added e.g. CIFARxxx., ImageNet.
Overall: I find that this paper needs to be improved both in terms of readability and technical contribution to ready for publication.


Modern natural language processing and understanding applications have enjoyed a great boost utilizing neural networks models. However, this is not the case for most languages especially low-resource ones with insufficient annotated training data. Cross-lingual transfer learning methods improve the performance on a lowresource target language by leveraging labeled data from other (source) languages, typically with the help of cross-lingual resources such as parallel corpora. In this work, we propose a zero-resource multilingual transfer learning model 1 that can utilize training data in multiple source languages, while not requiring target language training data nor cross-lingual supervision. Unlike most existing methods that only rely on language-invariant features for cross-lingual transfer, our approach utilizes both language-invariant and language-specific features in a coherent way. Our model leverages adversarial networks to learn language-invariant features and mixture-of-experts models to dynamically exploit the relation between the target language and each individual source language. This enables our model to learn effectively what to share between various languages in the multilingual setup. It results in significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging tasks including a large-scale real-world industry dataset.

This paper describes a model for cross lingual transfer with no target language information. This is a well written paper that makes a number of contributions:
1. It provides an interesting discussion of transfer form multiple source languages into a target language. This is a timely problem and the paper points out that adversarial networks may be too limiting in this setup.
2. It provides a modeling approach that deals with the limitations of adversarial networks as mentioned in (1).
3. It demonstrates the value of the proposed approach through an extensive experimental setup.
At the same time, I see two major limitations to the paper:
1. While the proposed approach is valid, it is not very original, at least in my subjective eyes. The authors integrate a classifier that combines the private, language-specific features so that not only features that are shared between all the involved languages can be used in the classification process. While this is a reasonable idea that works well in practice, IMO it is quite straight forward and builds on ideas that have been recently been proposed in many other works.
2. The authors claim that: "To our best knowledge, this work is the first to propose
an unsupervised CLTL framework without depending on any cross-lingual resource"
This is, unfortunately, not true. I refer the authors to the paper:
Deep Pivot-Based Modeling for Cross-language Cross-domain Transfer with Minimal Guidance. Yftah Ziser and Roi Reichart. EMNLP 2018.
In their lazy setup, the EMNLP authors do exactly that. They address the more complicated cross-language, cross-domain setup, but their model can be easily employed within a single domain. Their experiments even use the multilingual sentiment dataset used in the current paper. The model in the EMNLP paper shows to outperform adversarial networks, so it can be competitive here as well.


Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time. In this paper, we formulate a new task of unsupervised video-tovideo translation, which poses its own unique challenges. Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames. We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal 3D translator as an alternative solution to this problem. We evaluate our 3D method on multiple synthetic datasets, such as moving colorized digits, as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset. Our results show that frame-wise translation produces realistic results on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach, which is better able to learn the complex structure of video and motion and continuity of object appearance.

1) Summary
This paper proposes a 3D convolutional neural network based architecture for video-to-video translation. The method mitigates the inconsistency problem present when image-to-image translation methods are used in the video domain. Additionally, they present a study of ways to better setting up batched for the learning steps during networks optimization for videos, and also, they propose a new MRI-to-CT dataset for medical volumetric image translation. The proposed method outperforms the image-to-image translation methods in most measures.
2) Pros:
+ Proposed network architecture mitigates the pixel color discontinuity issues present in image-to-image translation methods.
+ Proposed a new MRI-to-CT dataset that could be useful for the community to have a benchmark on medical related research papers.
3) Cons:
Limited network architecture:
- The proposed neural network architecture is limited to only generate the number of frames it was trained to generate. Usually, in video generation / translation / prediction we want to be able to produce any length of video. I acknowledge that the network can be re-used to continue generating number of frames that are multiples of what the network was trained to generate, but the authors have not shown this in the provided videos. I would be good if they can provide evidence that this can be done with the proposed network.
Short videos:
- Another limitation that is related to the previously mentioned issue is that the videos are short, which in video-to-video translation, it should not be difficult to generate longer videos. It is hard to conclude that the proposed method will work for large videos from the provided evidence.
Lack of baselines:
- A paper from NVIDIA Research on video-to-video synthesis [1] (including the code)  came out about a month before the ICLR deadline. It would be good if the authors can include comparison with this method in the paper revision. Other papers such as [2, 3] on image-to-image translation are available for comparison. The authors simply say such methods do not work, but show no evidence in the experimental section. I peeked at some of the results in the papers corresponding websites, and the videos look consistent through time. Can the authors comment on this if I am missing something?
Additional comments:
The authors mention in the conclusion that this paper proposes “a new computer vision task or video-to-video translation, as well as, datasets, metrics and multiple baselines”. I am not sure that video-to-video translation is new, as it has been done by the papers I mention above. Maybe I am misunderstanding the statement? If so, please clarify. Additionally, I am not sure how the metrics are new. Human evaluation has been done before, the video colorization evaluation may be somewhat new, but I do not think it will generalize to tasks other than colorization. Again, If I am misunderstanding the statement, please let me know in the rebuttal.
4) Conclusion
The problem tackled is a difficult one, but other papers that are not included in experiments have been tested on this task. The proposed dataset can be of great value to the community, and is a clearly important piece of this paper. I am willing to change the current score if they authors are able to address the issues mentioned above.
References:
[1] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. "Video-to-Video Synthesis". In NIPS, 2018.
[2] Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz. Multimodal Unsupervised Image-to-Image Translation. In ECCV, 2018.
[3] Ming-Yu Liu, Thomas Breuel, Jan Kautz. Unsupervised Image-to-Image Translation Networks. In NIPS, 2017.


We study credit assignment problems in spatial multi-agent environments where agents pursue a joint objective. On the example of soccer, we rate the movements of individual players with respect to their potential for staging a successful attack. We propose a purely data-driven approach to simultaneously learn a model of agent movements as well as their ratings via an agent-centric deep reinforcement learning framework. Our model allows for efficient learning and sampling of ratings in the continuous action space. We empirically observe on historic soccer data that the model accurately rates agent movements w.r.t. their relative contribution to the collective goal.

This paper presents a method for learning the Q function from multi-agent demonstrations, such as trajectories of soccer players in a soccer game.  The basic ideas are:
-- The policy class pi(m|s) maps states s to actions m probabilistically.  In particular, the probability class is a mixture of Gaussians, with the mean/covariance functions and the mixture function all being instantiated via neural nets.  The behavior of the individual players are assumed to be conditionally independent given the current state.
-- A hand-designed reward function is used, such as the ball being in a "strike zone" of some kind for the team on offense.
-- The Q function class is also a neural net.
-- The policy pi is learned through maximum likelihood of the state/action pairs of the the pre-collected demonstrations.  This is essentially probabilistic behavioral cloning over the policy class.
-- The Q function is learned via Q-fitting over the demonstration data, i.e., minimizing squared error on the Bellman residual.  This requires the pre-specified reward function, the policy pi, and some details regarding the state transitions (e.g., the non-deterministic transitions of the ball).
-- The policy pi and the Q function are trained jointly by adding the two objectives.
-- The main empirical evaluation is comparing the quality of the Q function estimation against a naive baseline (random movements) and a non-mixture policy class (i.e., a single mode, rather than a mixture of Gaussians).  The results against the non-mixture policy class do not seem to show much difference in performance.
-- Some qualitative evaluations are also presented, but it's unclear what insight one is expected to gain from looking at them.
**Clarity**
The paper is reasonably well written.  Some aspects of the logical flow could be improved, but overall it's fine.  Detailed comments are:
-- The proposed approach is essentially imitation learning and not, strictly speaking, reinforcement learning.  Yet the work is positioned as reinforcement learning.  Perhaps it's just a difference in norms of terminology usage.
--  There are more multi-agent RL & IL works than what was discussed in the related works section.  Of course, an exhaustive discussion is not expected, but the writing makes it sound like there hasn't been much work at all.  Examples include:
https://arxiv.org/abs/1807.09936
https://arxiv.org/abs/1706.02275
http://www.cs.ox.ac.uk/people/shimon.whiteson/pubs/rashidicml18.pdf
-- The authors comment in the related work that team sports cannot yet be simulated.  This statement is sort of true, although it's not clear how true. For instance, the physics engines of many sports video games are very realistic.  Moreover, it doesn't strike me as the most prominent point of contrast with [Hausknecht & Stone 2016].  For instance, a more prominent point of contrast is simply imitation learning vs reinforcement learning.
-- It's not clear what the authors mean specifically when they refer to "average performance" contrasting with [Le et al. 2017b].  Also, desn't that work also condition on game context and situation?
-- Section 4.1 is written in a somewhat disorganized way.  The exposition alternates between discussing baselines and the evaluation methodology in a way that is confusing.
-- One of the baselines is described but then left to the appendix for analysis.  This doesn't make sense from a narrative perspective.
**Originality**
From a technical perspective, it's not clear that there's much novelty in this approach.  All the ingredients are pretty standard, and the ingredients are put together in a standard way.
From an applications perspective, there might be some novelty here but it's not clear.  The application of imitation learning to sports data is not new, but also not saturated either.  The specific idea of learning a better rating system is interesting, but it's not clear how fleshed out this application is in the paper (more comments on this below).
**Significance**
For me, the key question for significance boils down to: "Does this paper change the way people think about doing X?" for the most interesting instantiation of X possible.  In this case, it seems X should be computing a rating system for multi-agent spatial multi-agent behavior (as suggested by the title).  But the limitations in the model and the experiments make this a questionable proposition.
The model class all but ignores the multi-agent aspect (the agents are all independent conditioned on the states). Moreover there was no comparison against a model class that made a worse assumption on how to handle the multi-agent aspect.  So it's not clear how the multi-agent aspect is significant.
The evaluation of the rating approach is not convincing.  The approach essentially performed as well as a single-mixture baseline.  Furthermore, the direction V(s) learning approach is deferred to the appendix, and I'd like to see it included in the main paper.  Finally, it would be nice if there was some attempt to compare with a non-MDP based approach.  For instance, people directly compute things like P(goal|state).  I'd like to see evidence that this style of approach will outperform those approaches, which are the de facto standard in sports analytics right now.
**Overall Quality**
Based on the above comments, it is my opinion that this paper has too many holes in it to be ready for publication at a venue such as ICLR.  As a primarily applications paper, the burden is on the authors to demonstrate reasonably convincing evidence that this line of approach is valuable to the application.


ON THE TURING COMPLETENESS OF MODERN NEURAL NETWORK ARCHITECTURES
Alternatives to recurrent neural networks, in particular, architectures based on attention or convolutions, have been gaining momentum for processing input sequences. In spite of their relevance, the computational properties of these alternatives have not yet been fully explored. We study the computational power of two of the most paradigmatic architectures exemplifying these mechanisms: the Transformer (Vaswani et al., 2017) and the Neural GPU (Kaiser & Sutskever, 2016). We show both models to be Turing complete exclusively based on their capacity to compute and access internal dense representations of the data. In particular, neither the Transformer nor the Neural GPU requires access to an external memory to become Turing complete. Our study also reveals some minimal sets of elements needed to obtain these completeness results.

This paper seeks to answer the question of whether models which process sequences, but are not strictly classical RNNs, are Turing complete.
The authors present proofs that both the Transformer and Neural GPU are turing complete, under certain conditions. I do not consider myself qualified to properly verify the proof but it seems to be presented clearly. The authors note that the conditions involved are not how these models are used in the real world. Given the complex construction required for this more theoretically based proof, it seems reasonable that this should be published now, rather than waiting until the further work discussed in the final section is completed.
I have a number of questions where if a brief answer is possible, this would enhance the manuscript. The main question is, of the simplifications and approximations used for the proof, how much does that take the model away from what is used in practice? For example, the assumption of the piecewise linear sigmoid seems like a quite big change, as there are large regions of the space which now have zero gradients. If you run a real implementation of these models, with the normal sigmoid replaced by this one, does training still work? If not, what are the implications for the proof?
The rational numbers assumption is interesting - again I wonder how this would affect the model in reality, obviously all floating points on a computer represent rationals, but it would be interesting to get a better understanding on how the lack of infinite precision rationals on real hardware affects the main results.
Does the proof rely on the input and output dimensionality being the same? Eg in the preliminaries, x_i and y_i are both d-dimensional - could this be changed?
Overall this paper is novel and interesting, I have to give a slightly low confidence score because I'm unfamiliar with a lot of the background here (eg the Siegelamnn & Sontag work). The paper does seem concise and well written.
typos and minor points:
Circular convolution definition only appears to define the values directly adjacent to the border, would it be more appropriate to define S_{h+n, :, :} = S{n, :, :}?
paragraph above equation 5, 'vectores' -> 'vectors'


The gap between the empirical success of deep learning and the lack of strong theoretical guarantees calls for studying simpler models. By observing that a ReLU neuron is a product of a linear function with a gate (the latter determines whether the neuron is active or not), where both share a jointly trained weight vector, we propose to decouple the two. We introduce GaLU networks -networks in which each neuron is a product of a Linear Unit, defined by a weight vector which is being trained, with a Gate, defined by a different weight vector which is not being trained. Generally speaking, given a base model and a simpler version of it, the two parameters that determine the quality of the simpler version are whether its practical performance is close enough to the base model and whether it is easier to analyze it theoretically. We show that GaLU networks perform similarly to ReLU networks on standard datasets and we initiate a study of their theoretical properties, demonstrating that they are indeed easier to analyze. We believe that further research of GaLU networks may be fruitful for the development of a theory of deep learning.

The paper introduces a GaLU activation function, which is the product of a random gate function and a learnable linear function. The authors argue that empirically, neural networks with the GaLU activation is as effective as that with the ReLU activation, but theoretically, the GaLU activation is easier to understand because of the separation of the non-linearity and the learnable parameters. The the paper analyzes neural networks with one GaLU layer. Essentially, the network is a random transformation followed by a linear projection. This property enables analysis that are well known for the linear models.
Although the definition of GaLU is new, the idea of combining a non-linear projection with a linear transformation is an old one. [1] shows that many kernel SVM models can be written in this form. [2] [3] show that neural networks with various of activation functions can be relaxed to this form. However, these methods have never achieved performance that is as good as the state-of-the-art CNN models in challenging datasets (ImageNet or even CIFAR-10).
In section 3, the accuracies on MNIST (98%) and MNIST-fashion (88%) are quite low. They are not even as good as a classical kernel SVM, though the non-linear projection version of the kernel SVM has been well-studied in theory.
In section 4, the analyses are mostly standard for linear models and convex optimization. To the best of our knowledge, it doesn't introduce new insight on the understanding of non-convex optimization.
Overall, I think the paper and its theoretical analysis is built on an unsolid claim that the GaLU activation is a good replacement for traditional non-linear activation functions. The empirical study doesn't seem to support this claim. I cannot recommend accepting the paper.
[1] Random Features for Large-Scale Kernel Machines
[2] Learning Kernel-Based Halfspaces with the Zero-One Loss
[3] Convexified Convolutional Neural Networks


COMBINATORIAL ATTACKS ON BINARIZED NEURAL NETWORKS
Binarized Neural Networks (BNNs) have recently attracted significant interest due to their computational efficiency. Concurrently, it has been shown that neural networks may be overly sensitive to "attacks" -tiny adversarial changes in the input -which may be detrimental to their use in safety-critical domains. Designing attack algorithms that effectively fool trained models is a key step towards learning robust neural networks. The discrete, non-differentiable nature of BNNs, which distinguishes them from their full-precision counterparts, poses a challenge to gradient-based attacks. In this work, we study the problem of attacking a BNN through the lens of combinatorial and integer optimization. We propose a Mixed Integer Linear Programming (MILP) formulation of the problem. While exact and flexible, the MILP quickly becomes intractable as the network and perturbation space grow. To address this issue, we propose IProp, a decomposition-based algorithm that solves a sequence of much smaller MILP problems. Experimentally, we evaluate both proposed methods against the standard gradient-based attack (PGD) on MNIST and Fashion-MNIST, and show that IProp performs favorably compared to PGD, while scaling beyond the limits of the MILP.

The authors study the problem of generating strong adversarial attacks on binarized neural networks (networks whose weights are binary valued and have a sign function nonlinearity).  Since these networks are not continuous (due to the sign function nonlinearity), it is possible that standard gradient-based attack algorithms are not effective at producing adversarial examples. While this problem can be encoded as a mixed integer linear program, off-the-shelf MILP solvers are not scalable to larger/deeper networks. Thus, the authors propose a new target propagation style algorithm that attempts to infer desired activations at each layer (from the perspective of maximizing the adversary's objective) starting at the final layer and moving towards the input. The propagation at each layer requires solving another MILP (albeit a much smaller one). Further, in order to prevent the target propagation from discovering assignments at upper layers that are unachievable given the constraints at lower layers, the authors propose two heuristics (making small moves and penalizing deviations from the previous target values) to obtain an effective attack algorithm. The authors validate their approach experimentally on MNIST/Fashion MNIST image classifiers.
Quality: The paper is reasonably well written and the key ideas are communicated well. However, the experimental section needs to be improved significantly.
Clarity: The paper is easy to understand and organized well.
Originality: The application of target propagation in the context of adversarial examples is certainly novel and so are the specific enhancements proposed in the context of adversarial example generation. The
Significance: The study of adversarial examples for binarized networks is novel and important and effective attack generation algorithms are a significant first step towards training robust models of this type - this could enable deployment of robust and compact binarized classifiers in on-device settings (where model size is important).
Cons
My main concerns with this paper are regarding the experimental evaluation - I do not feel these are sufficient to justify the strength of the attack method proposed. Here are my broad concerns:
1. Even though the datasets used are small (MNIST/Fashion MNIST), the experimental validation of adversarial attacks is only performed on 100 test examples. This is not sufficiently representative (given experimental evidence with adversarial attacks on non-binarized models) and this needs to be addressed for the results to be considered conclusive.
2. The attack method is only compared to FSGM, which is known to be a rather poor attack even on non-binarized networks. The authors should compare to stronger gradient based attacks (like PGD) and gradient free attacks which have been used to break adversarial defenses that are nondifferentiable in prior work - https://arxiv.org/abs/1802.00420 and https://arxiv.org/abs/1802.05666). Further, the MILP approach used can be strengthened by doing better bound propagation (like in https://arxiv.org/pdf/1711.00455.pdf)
3. The attack radii used are very small compared to what has been used in non-binarized networks, where networks have been trained to even be verifiably robust to adversarial pertrubations of much larger radii (see for example https://arxiv.org/pdf/1805.12514.pdf). Given the existence of this work, it is important to evaluate the algorithms proposed on larger radii (since it is possible to construct non-binarized networks that are indeed robust to perburbations of eps=.1-.3 on MNIST).
4. Motivation for binarization: I assume that motivation for binarized models arising from faster training/inference times and smaller model sizes. However, to justify this, the authors need to compare their BNNs to comparable non-binarized neural networks (for example,ones that are similar  in terms of number of bits used to represent the model) on training time, inference time and adversarial robustness. Otherwise, it seems hard to see why binarized networks are valuable from a robustness.


Quantum mechanics fundamentally forbids deterministic discrimination of quantum states and processes. However, the ability to optimally distinguish various classes of quantum data is an important primitive in quantum information science. In this work, we trained near-term quantum circuits to classify data represented by quantum states using the Adam stochastic optimization algorithm. This is achieved by iterative interactions of a classical device with a quantum processor to discover the parameters of an unknown non-unitary quantum circuit. This circuit learns to simulate the unknown structure of a generalized quantum measurement, or positive-operator valued measure (POVM), that is required to optimally distinguish possible distributions of quantum inputs. Notably we used universal circuit topologies, with a theoretically motivated circuit design which guaranteed that our circuits can perform arbitrary input-output mappings. Our numerical simulations showed that quantum circuits could be trained to discriminate among various pure and mixed quantum states, exhibiting a trade-off between minimizing erroneous and inconclusive outcomes with comparable performance to theoretically optimal POVMs. We trained the circuit on different classes of quantum data and evaluated the generalization error on unseen quantum data. This generalization power hence distinguishes our work from standard circuit optimization and provides an example of quantum machine learning for a task that has inherently no classical analogue.

Summary of paper:
The authors partially integrate a neural network into classical approaches to classify the state of a quantum circuit. The model is not actually clear in what it is doing, but there are some trained weights somewhere. They allow for an "uncertainty" prediction by giving one more node than there are classification targets, corresponding to a less-penalized uncertain prediction. They evaluate their model on numerical simulations.
Strengths:
-
Weaknesses:
- The neural network architecture is entirely standard with nothing new.
- The paper is poorly written and very hard to follow.
- The focus is almost exclusively on the application, and yet the application is not explained effectively.
- The implication of the results and usefulness is not elaborated.
- The particular contributions are not clear.
Suggested Revisions:
- What is the 9b8d in the first sentence of the abstract?
- "...been developed to address the question [of] whether quantum mechanics..."
- "...for all the dataset[s] in Table 1..."


POINCARÉ GLOVE: HYPERBOLIC WORD EMBEDDINGS
Words are not created equal. In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal. In this paper, justified by the notion of delta-hyperbolicity or tree-likeliness of a space, we propose to embed words in a Cartesian product of hyperbolic spaces which we theoretically connect to the Gaussian word embeddings and their Fisher geometry. This connection allows us to introduce a novel principled hypernymy score for word embeddings. Moreover, we adapt the well-known Glove algorithm to learn unsupervised word embeddings in this type of Riemannian manifolds. We further explain how to solve the analogy task using the Riemannian parallel transport that generalizes vector arithmetics to this new type of geometry. Empirically, based on extensive experiments, we prove that our embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection. In particular, for word hypernymy, we obtain new state-of-the-art on fully unsupervised WBLESS classification accuracy.

The English, grammar and writing style is very good, as are the citations.
The technical quality appears to me to be very good (I am not an expert in Poincare spaces).
The authors demonstrate a good knowledge of the mathematical theory with the constructions made in Section 6.
The experimental write-up has been abbreviated.  The lexical entailment results Tables 6 and 7 are just sitting there without discussion, as far as I can see, as are the qualitative results Tables 4 and 5.   The entailment results are quite complex and really need supporting interpretation.  For instance, for Hyperlex, WN-Poincare is 0.512, above yours.
For your entailment score you say "For simplicity, we propose dropping the dependence in μ".  This needs more justification and discussion as it is counter-intuitive for those not expert in  Poincare spaces.
Section 6.2 presents the entailment score.  Note Nickel etal. give us a nice single formula.  You however, provide 4 paragraphs of construction from which an astute reader would then have to work on to extract your actual method.  I would prefer to see a summary algorithm given somewhere.  Perhaps you need another appendix.
RADAGRAD is discussed in Section 5, but I'd have preferred to see it discussed again in Section 8 and discussed to highlight what was indded done and the differences.  It certainly makes the paper non-reproducible.
A significant part of the theory in earlier sections is about the 50x2D method, but in experiments this doesn't seem to work as well.  Can you justify this some other how:  its much faster, its more interpretable?  Otherwise, I'm left thinking, why not delete this stuff?
The paper justifies its method with a substantial and winning comparison against vanilla GloVe.  That by itself is a substantial contribution.
But now, one is then hit with a raft of questions.  Embedding methods are popping up like daisies all over the fields of academia.  Indeed, word similarity and lexical entailment tasks themselves are proliferating too.  To me, its really unclear what one needs to achieve in the empirical section of a paper.  To make it worse, some folks use 500D, some 100D, some 50D, so results aren't always comparible.  Demonstrating one's work is state-of-the-art against all comers is a massive implementation effort.  I notice some papers now just compare against one other (e.g., Klami etal. ECML-PKDD, 2018).
My overall feeling is that this paper tries to compress too much into a small space (8 pages).
I think it really needs to be longer to present what is shown.   Moreover, I would want to see the inclusion of the work on 50x2D justified. So my criticisms are about the way the paper is written, not about the quality of the work.
Moroever, though, one needs to consider comparisons against models other than GloVe.
Addendum:  You know, what I really love about ICLR is the effort authors make to refresh their paper and respond to reviewers.  You guys did a great job.  Really impressed.  50x2D now clarified and some of the hasty/unexplained bits fixed.


In this paper, we propose a generalization of the BN algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. Batch normalization (BN) is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters. Our analysis can be easily generalized for original BN algorithm by setting some parameters to constant. To the best knowledge of authors, this analysis is the first of its kind for convergence with Batch Normalization introduced. We analyze a two-layer model with arbitrary activation function. The primary challenge of the analysis is the fact that some parameters are updated by gradient while others are not. The convergence analysis applies to any activation function that satisfies our common assumptions. For the analysis, we also show the sufficient and necessary conditions for the stepsizes and diminishing weights to ensure the convergence. In the numerical experiments, we use more complex models with more layers and ReLU activation. We observe that DBN outperforms the original BN algorithm on Imagenet, MNIST, NI and CIFAR-10 datasets with reasonable complex FNN and CNN models.

In this work, the authors propose a generalization of the batch normalization (BN) technique often used in training neural networks, and analyzed this convergence. In particular, a one hidden layer and one BN hidden layer fully connected network is considered, and a deterministic gradient descent algorithm with certain kind of BN has been considered in this work. The proposed “generalized” BN strategy is devised on the deterministic setting, but it is a slight generalization of the original BN by introducing a moving average operation. Classical results of Bertsekas is leveraged to show the asymptotic convergence of the algorithm.
I have the following three main comments about the paper.
1)	Only deterministic setting is considered, but in this case every time the entire data set will be used to perform the averaging, it appears to be much easier to analyze than the stochastic setting. Further the reviewer has doubt on whether the resulting deterministic algorithm has any practice value.
2)	Because the authors have used the Bertsekas/Tsitsiklis (B/T) argument, only asymptotic convergence is shown. It is not clear, even in the deterministic case, whether some kind of sublinear convergence rate can be obtained.
3)	Only one hidden layer of neural network with one BN operation is considered. It is not clear whether the analysis can be extended to multiple layers, despite the statement of the author saying that “the technique presented can be extended to more layers with additional notation”. In particular, when there are multiple layers, the BN layers will be further composite together across multiple nonlinear operations.
4)	The authors have mentioned that the derivative is always taking w.r.t. theta. However, in (9) is appears that the derivative is taken with respect to lambda, in order to get the Lipschitz condition on \lambda. This is a bit confusing. Also it is not clear how the gradient in Assumption 5 is defined.
5)	Assumption 5 does not make sense. Problem (1) is a constrained problem with both variables being confined in compact feasible sets. And this condition is important in Assumption2. Now the authors say that at stationary solution the gradient is zero? Please specify functions when this will happen. I will suggest that the authors use a proper definition of stationarity solution for constrained problems.
6)	Follow up on the previous point. The analysis builds upon B/T argument for unconstrained optimization. However it is not suitable for the constrained problems that the authors started out at the beginning of the paper. The authors may consider develop new analysis tools to understand the problem at hand, rather than assuming away the difficulties.


GENERALIZED TENSOR MODELS FOR RECURRENT NEURAL NETWORKS
Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency -a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN. Such networks, however, are not very often applied to real life tasks. In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency. Our theoretical results are verified by a series of extensive computational experiments.

The authors extend the theoretical results of a paper previously presented in the last edition of ICLR (2018), where it was demonstrated that Recurrent Neural Network can be interpreted as a tensor network decomposition based on the Tensor-Train (TT, Oseledets et al, 2011).
While previous results covered the multiplicative nonlinearity only, the contribution of the current paper is the extension of the analysis of universality and depth efficiency (Cohen et al, 2016) to different nonlinearities, for example ReLU (Rectified Linear Unit), which is very important from the practical point of view.
The paper is well written and have a good structure. However, I found that some deep concepts are not well introduced, and maybe other more trivial results are discussed with unnecessary details. The following comments could help authors to improve the quality of presentation of their paper:
-	Section 3.1 (Score Functions and Feature Tensor) is a bit short and difficult to read.
o	Maybe, a more motivating introduction could be included in order to justify the definition of score functions (eq. 2).
o	It would be also nice to state that, according to eq. (3), the feature tensor is a rank-1 tensor.
o	I would suggest moving the definition of outer product to the Appendix, since most readers know it very well.
o	It is said that eq. 2 possesses the universal approximation property (it can approximate any function with any prescribed precision given sufficiently large M). It is not clear which is the approximation function.
-	A Connection with Tensor-Ring (TR) format, if possible, could be helpful: It is known that TR format (Zhao et al, 2016, arXiv:1606.05535), which is obtained by connecting the first and last units in a TT model, helps to alleviate the requirement of large ranks in the first and last the core tensors of a TT model reaching to a decomposition with an evenly distributed rank bounds. I think, it would be interesting to make a connection of RNN to TR because the assumption of R_i < R for all i becomes more natural. I would like to see at least some comment from the authors about the applicability of TR in the context of analysis of RNN, if possible. Maybe, also the initial hidden state defined in page 5 can be avoided if TR is used instead of TT.
-	Fig 2 shows that Test accuracy of a shallow network (CP based) is lower and increases with the number of parameters approaching to the one for RNN (TT based). It would be necessary to show the results for an extended range in the number of parameters, for example, by plotting the results up to 10^6. It is expected that, at some point, the effect of overfitting start decreasing the test accuracy.
-       When scores functions are presented (eq. 2) it is written the term "logits" between brackets. Could you please clarify why this term is introduced here? Usually, logit of a probability p is defined as L(p)=p/(1-p). What is the usage of this term in this work?
-      I think the theory is presented for a model with the two-classes only but used for multiple classes in the experimental sections. It should be necessary to make some comment about this in the paper.
-      Details about how the RNN based on TT is applied must be added. More specifically, the authors should provide answers to clarify the following questions:
(i) Are patches overlapped or non-overlapped?
(ii) What value of M is used? and is there any general rule for this choice?
(iii) How the classification in the 10-classes is obtained? Are you using a softmax function in the last layer? Are you using one weight tensor W_c per class (c=1,2,...,10). Please provide these technical details.
(iv) Please, specify which nonlinear activation sigma is used in the feature map f_\theta(x).
(v) How many feature maps are used? and, Are the matrix A and vector b learned from training dataset or only the TT-cores need to be learned?


Counterfactual Regret Minimization (CRF) is a fundamental and effective technique for solving Imperfect Information Games (IIG). However, the original CRF algorithm only works for discrete state and action spaces, and the resulting strategy is maintained as a tabular representation. Such tabular representation limits the method from being directly applied to large games and continuing to improve from a poor strategy profile. In this paper, we propose a double neural representation for the imperfect information games, where one neural network represents the cumulative regret, and the other represents the average strategy. Furthermore, we adopt the counterfactual regret minimization algorithm to optimize this double neural representation. To make neural learning efficient, we also developed several novel techniques including a robust sampling method, mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) and Monte Carlo Counterfactual Regret Minimization Plus (MCCFR+) which may be of independent interests. Experimentally, we demonstrate that the proposed double neural algorithm converges significantly better than the reinforcement learning counterpart.

========= Summary =========
The authors propose "Double Neural CFR", which uses neural network function approximation in place of the tabular update in CFR. CFR is the leading method for finding equilibria in imperfect information games. However it is typically employed with a tabular policy, limiting its applicability large games. Typically, hand-crafted abstractions are employed for games that are too large for exact tabular solutions. Function approximation could remove the necessity for hand-crafted abstractions and allow CFR to scale to larger problems.
The DN-CFR algorithm roughly consists of:
- start with an arbitrary vmodel_0, smodel_0
for t = 0,1,2,3:
  - collect a "batch" of (infoset I, immediate counterfactual value v_I) samples by traversal against vmodel_t (as well as I, strategy samples)
  - train a new network vmodel_{t+1} on this "batch" with y=(v_I + vmodel_t(I)) and MSE loss
  - similarly for (I, strategy)
- return smodel_t
The authors also propose a novel MC samping strategy this is a mixture between outcome and external sampling.
DN-CFR is evaluated on two games: a variant of Leduc hold-em with stack size 5, and one-card poker with 5 cards. If I understand correctly, these games have <10,000 and <100 infosets, respectively. The authors show that DN-CFR achieves similar convergence rates to tabular CFR, and outperform NFSP variants.
========== Comments ========
The authors are exploring an important problem that is of great interest to the IIG community. Their application of NN function approximation is reasonable and mostly theoretically well-grounded (but see below), I think it's on the right track. However, the games that are used for evaluation are very small, in fact I believe they have fewer states than the number of parameters in their network (the number of network parameters is not provided but I assume >1000). As a result, the NN is not providing any compression or generalization, and I would expect that the network can memorize the training set data exactly, i.e. predict the exact mean counterfactual value for each infoset over the data. If that's true, then DN-CFR is essentially exactly replicating tabular CFR (the approximation serves no purpose).
As a result, in my opinion this work fails to address the important challenges for function approximation in CFR, namely:
- Can function approximation allow for *generalization across infosets* in order to reduce sample complexity of CFR (i.e. an unsupervised abstraction)? Are specific network architectures required for good generalization?
- The magnitude of counterfactual regrets in the support of the equilibrium decays to zero relative to dominated actions. Are NN models capable to estimating the regrets accurately enough to converge to a good strategy?
- Are optimization methods able to deal with the high variance in large IIGs?
- Since each successive approximation is being trained from the previous NN, does this cause errors to accumulate?
- How do approximation errors accumulate across CFR iterations?
- Is minimizing MSE loss sufficient to approximate the strategy well? (since the mapping from regrets -> strategy is non-linear)
I believe there is also a theoretical problem with this algorithm. In Eq. 8, they minimize the loss of CF value predictions *over the distribution of infosets in the last CFR step ("batch")*. However, this distribution may change between CFR iterations, for example if the strat_t folds 2-3 on the preflop then flop infosets with 2-3 hole cards will never be observed on iteration t+1. As a result, the NN loss will only be minimized over infosets observed in the last iteration - so the network will "forget" the regrets for all other infosets. I think this issue does not arise in these toy games because all infosets are observed at each iteration, but this is certainly not the case in real games.
There are a number of ways that this issue could be addressed (e.g. train on historical infosets rather than the current batch, etc.) These would need to be explored.
I would recommend that the authors evaluate on more complex games to answer the important questions stated above and resubmit. I think this would also make an excellent workshop submission in its current state as it contains many interesting ideas.
Detailed comments:
"... the original CFR only works for discrete stand and action spaces...": Are the authors implying that DN-CFR addresses this limitation?
"Moravk" -> "Moravcik"
"...these methods do not explicitly take into account the hidden information in a game..." Could you clarify? Is your point that these methods operate on the normal form rather than extensive form game?
"care algorithm design" -> "careful algorithm design"
The paragraph starting "In standard RNN..." should be reduced or moved to appendix. The exact NN architecture is not central to the ideas, and there are no experimental comparison with other architectures so we have no evidence that the architecture is relevant.


Humans learn to play video games significantly faster than the state-of-the-art reinforcement learning (RL) algorithms. People seem to build simple models that are easy to learn to support planning and strategic exploration. Inspired by this, we investigate two issues in leveraging model-based RL for sample efficiency. First we investigate how to perform strategic exploration when exact planning is not feasible and empirically show that optimistic Monte Carlo Tree Search outperforms posterior sampling methods. Second we show how to learn simple deterministic models to support fast learning using object representation. We illustrate the benefit of these ideas by introducing a novel algorithm, Strategic Object Oriented Reinforcement Learning (SOORL), that outperforms state-of-the-art algorithms in the game of Pitfall! in less than 50 episodes.

-- Summary --
The paper proposes to learn (transition) models (for MDPs) in terms of objects and their interactions. These models are effectively deterministic and are compatible with algorithms for planning with count-based exploration. The paper demonstrates the performance of one such planning method in toy tasks and in Pitfall, as well as a comparison with other planning methods in the toy tasks. The proposed model-based method, called SOORL, yields agents that perform better on Pitfall with a small amount of data.
-- Assessment --
As a positive, the results of the paper are favorable compared to previous work, with good sample efficiency, and they demonstrate the viability of the proposed approach. The most negative point is that SOORL relies on limiting domain-specific biases that are hard to remove or circumvent.
-- Clarity --
The paper is somewhat clear. There are many typos and mistakes in writing, and at parts (for example, the second paragraph of Section 4.2) the explanations are not clear.
-- Originality --
I believe the work is original. The paper explores a natural idea and the claims/results are not surprising, but as far as I am aware it has not been tried before.
-- Support --
The paper provides support for some of the claims made. The comparison to related work contains unsupported claims ("we studied how imperfect planning can affect exploration") and could be more upfront about the weaknesses of the proposed method. The claims in the introduction are sufficiently supported.
-- Significance --
It would be hard to scale SOORL to other tasks, so it is unlikely to be adopted where end-to-end learning is wanted. Therefore I believe the impact of the paper to be limited.
There is also the question of whether the paper will attract interest and people will work on addressing the limitations of SOORL. I would like to hear more from the authors on this point.
-- For the rebuttal --
My greatest doubt is whether the paper will attract enough interest if published, and it would be helpful to hear from the authors on why they think future work will build on the paper. Why is the proposed approach a step in the right direction?
-- Comments --
Sample efficiency: The paper should be more clear about this point. It seems that 50 episodes were used for getting the positive reward in Pitfall, which is great.
Object detection: I am happy with the motivation about how we can remove the hand-made object detection. It is important the other strong assumptions (object interaction matrix, for example) can be removed as well. My opinion on simplifications is this: They are ok if they are being used to make experiments viable and they can be removed when scaling up; but they are not ok if there is no clear way to remove them.
Known interaction matrix: It may be possible to remove this requirement using the tools in [1]
Deterministic model: The use of no-ops to make the model deterministic seems right if the ultimate goal is to make the model deterministic, but it seems unsuited if the model is to be used for control. Maybe the model needs to be temporally extended as I thought the paper was proposing in Section 4.2 but Section 4.3 suggests that this temporal extension was not a good idea. Is my understanding correct?
Exploration: I was a bit confused about how the text discusses exploration. UCT uses OFU, but the text suggests that it does not. What are the components for exploration? Both a bonus on unseen transitions and the confidence interval bonus? Also, the paper would have to provide support for the claim that "with limited number of rollouts, the agent might not observe the optimistic part of the model, in contrast to optimistic MCTS where optimism is build into every node of the tree". However, it is fair to say that in the to domains MCTS seemed has performed better, and for that reason it has been chosen instead of Thompson Sampling for the later experiments.
Writing: The paper has a number of typos and mistakes that need to be fixed. To point out a few:
* I would suggest more careful use of "much" and "very"
* For citations, "Diuk et al. (2008) also proposed..." and "(UCT, Kocsis & Szepesvari, 2006)"
Claims: I think the claims made in the introduction could be stated more clearly in the conclusion. (Intro) "We show how to do approximate planning" --> (Conclusion) "Our model learning produces effectively deterministic models that can then be used by usual planning algorithms".
-- References --
[1] Santoro et al., 2017. "A simple neural network module for relational reasoning"