Hierarchical neural architectures can efficiently capture long-distance dependencies and have been used for many document-level tasks such as summarization, document segmentation, and sentiment analysis. However, effective usage of such a large context can be difficult to learn, especially in the case where there is limited labeled data available. Building on the recent success of language model pretraining methods for learning flat representations of text, we propose algorithms for pre-training hierarchical document representations from unlabeled data. Unlike prior work, which has focused on pre-training contextual token representations or context-independent sentence/paragraph representations, our hierarchical document representations include fixed-length sentence/paragraph representations which integrate contextual information from the entire documents. Experiments on document segmentation, document-level question answering, and extractive document summarization demonstrate the effectiveness of the proposed pre-training algorithms.

In this work, the authors explore different ways to pre-train contextualized word and sentence representations for use in other tasks. They propose two main methods: a straight-forward extension of the ElMO model for hierarchical uni-directional language models, and a de-noising auto-encoder type method which allows to train bi-directional representations. The learned contextual representations are evaluated on three downstream tasks, demonstrating the superiority of the bi-directional training setting, and beating strong baselines on extractive summarization.
The method is clearly presented and easy to follow, and the experiments do seem to support the author's claims, but their exposition misses several important details (or could be presented more clearly). For the document segmentation task, are the articles taken from a held-out set, or are they contained in the pre-training set? For passage retrieval, is the representation the same or are the representations re-trained from scratch using paragraph blocks? What exactly are the other features (those can go in the appendix)? And for the extractive summarization task, how many sentences are selected? Is pre-training also done on Wikipedia, or are those representations trained on news text?
A comparison to non-contextualized sentence representations would also be welcome (SkipThought, InferSent, ElMO-pool for settings other than passage retrieval). Note also that the local pre-training is not equivalent to ElMO, as the later sees context form the whole document rather than just the current sentence.
It is interesting to see that contextualized sentence representations can be used and that the Mask-LM objective yields better results than L+R-LM, but these points would be better made if the above questions were answered.


The formation of structure in the brain, that is, of the connections between cells within neural populations, is by large an unsupervised learning process: the emergence of this architecture is mostly self-organized. In the primary visual cortex of mammals, for example, one may observe during development the formation of cells selective to localized, oriented features. This leads to the development of a rough representation of contours of the retinal image in area V1. We modeled these mechanisms using sparse Hebbian learning algorithms. These algorithms alternate a coding step to encode the information with a learning step to find the proper encoder. A major difficulty faced by these algorithms is to deduce a good representation while knowing immature encoders, and to learn good encoders with a non-optimal representation. To address this problem, we propose to introduce a new regulation process between learning and coding, called homeostasis. Our homeostasis is compatible with a neuro-mimetic architecture and allows for the fast emergence of localized filters sensitive to orientation. The key to this algorithm lies in a simple adaptation mechanism based on non-linear functions that reconciles the antagonistic processes that occur at the coding and learning time scales. We tested this unsupervised algorithm with this homeostasis rule for a range of existing unsupervised learning algorithms coupled with different neural coding algorithms. In addition, we propose a simplification of this optimal homeostasis rule by implementing a simple heuristic on the probability of activation of neurons. Compared to the optimal homeostasis rule, we show that this heuristic allows to implement a more rapid unsupervised learning algorithm while keeping a large part of its effectiveness. These results demonstrate the potential application of such a strategy in machine learning and we illustrate this with one result in a convolutional neural network.

Please consider this rubric when writing your review:
1. Briefly establish your personal expertise in the field of the paper.
2. Concisely summarize the contributions of the paper.
3. Evaluate the quality and composition of the work.
4. Place the work in context of prior work, and evaluate this work's novelty.
5. Provide critique of each theorem or experiment that is relevant to your judgment of the paper's novelty and quality.
6. Provide a summary judgment if the work is significant and of interest to the community.
1. I am a researcher working at the intersection of machine learning,
computational neuroscience and biological vision.  I have experience
with neural network models and visual neurophysiology.
2. This paper develops and tests an adaptive homeostatic algorithm for
unsupervised visual feature learning (for example for learning models
of early visual processing/V1).
3.The work spends a lot of pages describing the general problem of
unsupervised feature learning and the history of the base algorithms.
The literature review is quite extensive.  The new content appears to
be in section 2.2 (Histogram Equalization Homeostasis - HEH), where a
simple idea to keep all units with balanced activity over the set of
natural images.  The authors also develop a computationally cheaper
version they call HAP (Homeostasis on Activation Probability) The
authors show that their F function is optimized quicker with the HEH
and HAP algorithms.  I would like to see how these curves vary with
the number of neurons (e.g. can you add X% more neurons and get
similar convergence speed -- and if so which is more computationally
costly)?
4. Many groups have developed various homeostatic algorithms for
unsupervised learning, though I have not seen this exact one before.
5.  The experiments reveal the resulting receptive fields and show the
decrease in the F function (error function).    The resulting receptive fields
do not seem that different to me between the different methods.  I am also not
that convinced that the faster convergence as a function of learning step is that important
especially as the learning steps may be more computationally expensive for this method.
6. I am not sure how interesting this work will be for the ICLR audience,
as it is not clear how important the faster convergence and more even
utilization of neurons is (and how it would compare computationally
with just having more neurons).


We address the problem of imitating multi-modal expert demonstrations in sequential decision making problems. In many practical applications, for example video games, behavioural demonstrations are readily available that contain multi-modal structure not captured by typical existing imitation learning approaches. For example, differences in the observed players' behaviours may be representative of different underlying playstyles. In this paper, we use a generative model to capture different emergent playstyles in an unsupervised manner, enabling the imitation of a diverse range of distinct behaviours. We utilise a variational autoencoder to learn an embedding of the different types of expert demonstrations on the trajectory level, and jointly learn a latent representation with a policy. In experiments on a range of 2D continuous control problems representative of Minecraft environments, we empirically demonstrate that our model can capture a multi-modal structured latent space from the demonstrated behavioural trajectories.

This paper presents an approach to multi-modal imitation learning by using a variational auto-encoder to embed demonstrated trajectories into a structured latent space that captures the multi-modal structure. This is done through a stochastic neural network with a bi-directional LSTM and mean pooling architecture that predicts the mean and log-variance of the latent state. This is followed by a state and action/policy decoder (both LSTMs) that recursively generate trajectories from latent space samples. The entire model is trained by optimising the ELBO on a set of pre-specified expert demonstrations. At test time, samples are generated from the latent space and recursively decoded to generate state and action trajectories. The method is tested on three low-dimensional continuous control tasks and is able to learn structured latent spaces capturing the modes in the training data as well as generating good trajectory reconstructions.
Learning from multi-modal demonstration data is an important sub-area in imitation learning. As the paper pointed out, there has been a lot of recent work in this area. A lot of the ideas in this paper are similar to those proposed in prior work -- the network for embedding the trajectory is similar to the ones from Wang et al & Co-Reyes et al with the major difference being in the structure of the action decoder (and what inputs to encoder). Also, prior work has dealt with problems that are high-dimensional (Wang et al) and has shown results when operating directly on visual data (InfoGAIL). Comparatively, the results in this paper are on toy problems.
As there is no direct comparison to prior work provided in the paper, it is hard to quantify how much better the proposed approach is in comparison to prior work. For example, the "2D Circle Example" was taken from the InfoGAIL paper. It would have been good to use that as a baseline example to compare those two methods and highlight the advantages of the proposed approach -- did it require less data? fewer environment interactions? etc.
The results on the Zombie Attack Scenario seem poor. Specifically, in the avoid scenario, the approach seems to fail almost half the time. It would be good if the authors spend more time on this -- again, a comparison to prior work would establish some baselines and give us a good idea of the expected performance on this scenario. The videos show a single representative example for the "Attack" and "Avoid" scenarios. More examples including failures need to be included so that the distribution of results can be captured.
There is little in terms of generalisation or ablation studies in the paper. For example, in the Zombie Attack Scenario one could generate data with different zombie behaviours and measure performance on held out behaviours. Similarly, as an ablation, the authors could look at directly predicting actions instead of states & actions (states could be generated through a pre-trained dynamics model).
Figure 6. is hard to parse and could be explained better. No details are provided on the network architecture (number/size of the LSTM/fully connected layers), number of demonstrations used, training algorithm, hyper-parameters etc.
Few typos in the paper:
  Page 6 - between the animation links 'avoiding' 'region'
  Fig 7 caption - the zombie but are not in attacking range -> but the zombies are not in the attacking range,
Relevant citations that can be added:
1) Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G., & Lim, J. J. (2017). Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems (pp. 1235-1245).
2) Tamar, A., Rohanimanesh, K., Chow, Y., Vigorito, C., Goodrich, B., Kahane, M., & Pridmore, D. (2018). Imitation Learning from Visual Data with Multiple Intentions.
Overall, I find the paper to be incremental and lacking good experimental results and comparisons. The strengths of the paper are not clear and need to be explained and evaluated well. Substantial work is needed to significantly improve the paper before it can be accepted.


HIERARCHICAL RL USING AN ENSEMBLE OF PROPRIOCEPTIVE PERIODIC POLICIES
In this work we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks. The agent is split into a low-level and a high-level policy. The low-level policy only accesses internal, proprioceptive dimensions of the state observation. The low-level policies are trained with a simple reward that encourages changing the values of the non-proprioceptive dimensions. Furthermore, it is induced to be periodic with the use a "phase function." The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time. Using this approach, we solve difficult maze and navigation tasks with sparse rewards using the Mujoco Ant and Humanoid agents and show improvement over recent hierarchical methods.

Brief summary:
HRL method which uses a 2 level hierarchy for sparse reward tasks. The low level policies are only provided access to proprioceptive parts of the observation, and are trained to maximize change in the non-proprioceptive part of the state as reward. The higher level policy is trained as usual by commanding lower level policies.
Overall impression:
I think the paper has a major assumption about the separation of internal and external state, thereby setting the form of the low level primitives. This may not be fully general, but is particularly useful for the classes of tasks shown here as seen from the strong results. I would like to see the method applied more generally to other robotic tasks, and a comparison to Florensa et al. And perhaps the addition of a video which shows the learned behaviors.
Introduction:
the difficulty of learning a high-level controller when the low-level policies shifts -> look at “data efficient hierarchical reinforcement learning” (Nachum et al)
The basic assumption that we can separate out observations into proprioceptive and not proprioceptive can often be difficult. For example with visual inputs or entangled state representations, this might be very challenging to extract. This idea seems to be very heavily based on what is “internal” and what is “external” to the agent, which may be quite challenging to separate.
The introduction of phase functions seems to be very specific to locomotion?

Related work:
The connection of learning diverse policies should be discussed with Florensa et al, since they also perform something similar with their mutual information term. DeepMimic, DeepLoco (Peng et al) also use phase information in the state, worthwhile to cite.
Section 3.1:
The pros and cons of making the assumption that representation is disentangled enough to make this separation, should be discussed.
Also, the internal and external state should be discussed with a concrete example, for the ant for example.
Section 3.2:
The objective for learning diverse policies is in some sense more general than Florensa et al, but in the same vein of thinking. What are the pros and cons of this approach over that?

The objective is greedy in the change of external state. We’d instead like something that over the whole trajectory maximizes change?
Section 3.3:
How well would these cyclic objectives work in a non-locomotion setting? For example manipulation
Section 3.4:
This formulation is really quite standard in many HRL methods such as options framework. The details can be significantly cut down, and not presented as a novel contribution.
Experiments:
It is quite cool that Figure 2 shows very significant movement, but in some sense this is already supervised to say “move the CoM a lot”. This should be compared with explicitly optimizing for such an objective, as in Florensa et al. I’m not sure that this would qualify as “unsupervised” per se. As in it too is using a particular set of pre-training tasks, just decided by the form of choosing internal and external state.
all of the baselines fail to get close to the goal locations.-> this is a bit surprising? Why are all the methods performing this poorly even when rewarded for moving the agent as much as possible.
Overall, the results are pretty impressive. A video would be a great addition to the paper.
Comparison to Eysenbach et al isn’t quite fair since that method receives less information. If given the extra information, the HRL method performs much better (as indicated by the ant waypoint plot in that paper).


Towards Metamerism via Foveated Style Transfer
The problem of visual metamerism is defined as finding a family of perceptually indistinguishable, yet physically different images. In this paper, we propose our NeuroFovea metamer model, a foveated generative model that is based on a mixture of peripheral representations and style transfer forward-pass algorithms. Our gradient-descent free model is parametrized by a foveated VGG19 encoder-decoder which allows us to encode images in high dimensional space and interpolate between the content and texture information with adaptive instance normalization anywhere in the visual field. Our contributions include: 1) A framework for computing metamers that resembles a noisy communication system via a foveated feed-forward encoder-decoder network -We observe that metamerism arises as a byproduct of noisy perturbations that partially lie in the perceptual null space; 2) A perceptual optimization scheme as a solution to the hyperparametric nature of our metamer model that requires tuning of the image-texture tradeoff coefficients everywhere in the visual field which are a consequence of internal noise; 3) An ABX psychophysical evaluation of our metamers where we also find that the rate of growth of the receptive fields in our model match V1 for reference metamers and V2 between synthesized samples. Our model also renders metamers at roughly a second, presenting a ×1000 speed-up compared to the previous work, which allows for tractable data-driven metamer experiments.

Summary:
The paper proposes a fast method for generating visual metamers – physically different images that cannot be told apart from an original – via foveated, fast, arbitrary style transfer. The method achieves the same goal as an earlier approach (Freeman & Simoncelli 2011): locally texturizing images in pooling regions that increase with eccentricity, but is orders of magnitude faster. The authors perform a psychophysical evaluation to test how (in)discriminable their synthesized images are amongst each other and compared with originals. Their experiment replicates the result of Freeman & Simoncelli of a V2-like critical scaling in the synth vs. synth condition, but shows that V1-like or smaller scaling is necessary for the original vs. synth condition.
I reviewed an earlier version of this paper for a different venue, where I recommended rejection. The authors have since addressed some of my concerns, which is why I am more positive about the paper now.
Strengths:
+ The motivation for the work is clear and the implementation straightforward, combining existing tools from style transfer in a novel way.
+ It's fast. Rendering speed is indeed a bottleneck in existing methods, so a fast method is useful.
+ The perceptual quality of the rendered images is quantified by psychophysical testing.
+ The role of the scaling factor for the pooling regions is investigated and the key result of Freeman & Simoncelli (pooling regions scale with 0.5*eccentricity) is replicated with the new method. In addition, the result of Wallis et al. (2018) that lower scale factors are required for original vs. synth is replicated as well.
Weaknesses:
- Compared with earlier work, an additional fudge parameter (alpha) is introduced. It is not clear why it is necessary and it complicates interpretation.
- The paper contains a number of sections with obscure mathiness and figures that I can't follow and whose significance is unclear.
Conclusion:
The work is well motivated, the method holds up to its promise of being fast and is empirically validated. However, it feels quite ad-hoc and the writing of the paper is very obscure at various places, which leaves room for improvement.
Details:
- The motivation for introducing alpha not clear to me. Wasn't the idea of F&S that you can reduce the image to its summary statistics within a pooling region whose size scales with eccentricity? Why do you need to retain some content information in the first place? How do images with alpha=1 (i.e. keep only texture) look?
- Related to above, why does alpha need to change with eccentricity? Experiment 1 seems to suggest that changing alpha leads to similar SSIM differences between synths and originals as F&S does, but what's the evidence that SSIM is a useful/important metric here?
- Again related to above, why do you not use the same approach of blending pooling regions like F&S did instead of introducing alpha?
- I would like to know some details about the inference of the critical scaling. It seems surprisingly spot on 0.5 as in F&S for synth vs. synth, but looking at the data in Fig. 12 (rightmost panel), I find the value 0.5 highly surprising given that all the blue points lie more or less on a straight line and the point at a scaling factor of 0.5 is clearly above chance level. Similarly, the fit for original vs. synth does not seem to fit the data all that well and a substantially shallower slope seems equally plausible given the data. How reliable are these estimates, what are the confidence intervals, and was a lapse rate included in the fits (see Wichmann & Hill 2001)?
- I don't get the point of Figs. 4, 13 and 14. I think they could as well be removed without the paper losing anything. Similarly, I don't think sections 2.1 and the lengthy discussion (section 5) are useful at all. Moreover, section 3 seems bogus. I don't understand the arguments made here, especially because the obvious options (alpha=1 or overlapping pooling regions; see above) are not even mentioned.
- How is the model trained? Do the authors use the pre-trained model of Huang & Belongie or is the training different in the context of the proposed method? I could only find the statement that the decoder is trained to invert the encoder, but that doesn't seem to be what Huang & Belongie's model does and the paper does not say anything about how it's trained to invert. Please clarify.
- At various places the writing is somewhat sloppy (missing words, commas, broken sentences), which could have been avoided by carefully proof-reading the paper.


The input method is an essential service on every mobile and desktop devices that provides text suggestions. It converts sequential keyboard inputs to the characters in its target language, which is indispensable for Japanese and Chinese users. Due to critical resource constraints and limited network bandwidth of the target devices, applying neural models to input method is not well explored. In this work, we apply a LSTM-based language model to input method and evaluate its performance for both prediction and conversion tasks with Japanese BCCWJ corpus. We articulate the bottleneck to be the slow softmax computation during conversion. To solve the issue, we propose incremental softmax approximation approach, which computes softmax with a selected subset vocabulary and fix the stale probabilities when the vocabulary is updated in future steps. We refer to this method as incremental selective softmax. The results show a two order speedup for the softmax computation when converting Japanese input sequences with a large vocabulary, reaching real-time speed on commodity CPU. We also exploit the model compressing potential to achieve a 92% model size reduction without losing accuracy.

The paper demonstrates the main challenge of using LSTM-based language models for input method in real time is the huge amount of computation in the softmax. The authors present a system to speed up the inference by avoiding computing the full softmax in the Japanese conversion task, where the number of output words can be limited from the mapping of the input sequence through a lexicon. The experiment result is encouraging in that the proposed incremental selective softmax approach significantly reduces latency over the standard inference with the full softmax computation while not hurting accuracy much. The paper also evaluates the effect of quantization for LSTM LM model compression in terms of size and accuracy.
However, there are a few major problems of the paper as follows:
1. The main weakness in the experiment setup is that it misses a few competitive baselines in terms of inference speed, notably hierarchical softmax[1] and self normalization[2]. In the Japanese conversion task in the paper, it only needs to evaluate the scores of limited output words that are given from the mapping of the input sequence through the lexicon. This is exactly like the rescoring setup in speech recognition and machine translation, where self-normalization is typically used for efficient inference to avoid computing the expensive softmax normalization term [2,3]. Assuming the number of selected output words is K and the entire vocabulary size is V, then the time complexity is O(K logV) for the hierarchical softmax, O(K) for self normalization, but O(V) for all the baselines in the paper. Self normalization is simple to implement and works well in practice, while the proposed incremental selective softmax approach in the paper needs an additional step to sample most frequent words to adjust the normalization term. Without showing the self normalization result, I am not convinced that the proposed approach is better and needed.
[1] F. Morin and Y. Bengio. "Hierarchical Probabilistic Neural Network Language Model," in Proc. of AISTATS, 2005,
[2] J. Devlin et al., "Fast and Robust Neural Network Joint Models for Statistical Machine Translation," in Proc. of ACL, 2014.
[3] Y. Shi, W. Zhang, M. Cai and J. Liu, "VARIANCE REGULARIZATION OF RNNLM FOR SPEECH RECOGNITION," in Proc. of ICASSP, 2014.
2. The proposed approach would only be useful in speeding up the conversion task, but not applicable to the prediction task where it needs to evaluate all words and choose the top hypotheses. Also how is the latency of the prediction task compared to conversion task? Please also add it to the experiment result.
3. The idea of using quantization for neural network model compression is not novel (even for language model), although it is listed as one of the main contributions in Section 1.
So in general, I think the paper is insufficient in novelty and missing competitive baselines.
Some specific comments:
4. Figure 2(b) is not clear what it means, and not referenced anywhere in the paper.
5. The last 3 lines in Section 3: "as each path has different missing vocabularies": Why is that? The candidates of the output words should only depend on the input sequence and the lexicon, based on Eq(1)(2).
6. It is not clear how to adjust the probability in the second pass of incremental selective softmax. The description "we compute a union of all missing vocabularies, and then recompute the logits of them in batch." is unclear what it means.
7. Section 4.2: "is measure with numpy" -> "is measured with numpy".
8. Section 4.4: It is not clean how "76x speedup" is computed from Table 2 since all the time numbers are rounded. Consider also showing one digit after the decimal point.


Many techniques are developed to defend against adversarial examples at scale. So far, the most successful defenses generate adversarial examples during each training step and add them to the training data. Yet, this brings significant computational overhead. In this paper, we investigate defenses against adversarial attacks. First, we propose feature smoothing, a simple data augmentation method with little computational overhead. Essentially, feature smoothing trains a neural network on virtual training data as an interpolation of features from a pair of samples, with the new label remaining the same as the dominant data point. The intuition behind feature smoothing is to generate virtual data points as close as adversarial examples, and to avoid the computational burden of generating data during training. Our experiments on MNIST and CIFAR10 datasets explore different combinations of known regularization and data augmentation methods and show that feature smoothing with logit squeezing performs best for both adversarial and clean accuracy. Second, we propose an unified framework to understand the connections and differences among different efficient methods by analyzing the biases and variances of decision boundary. We show that under some symmetrical assumptions, label smoothing, logit squeezing, weight decay, mix up and feature smoothing all produce an unbiased estimation of the decision boundary with smaller estimated variance. All of those methods except weight decay are also stable when the assumptions no longer hold.

The paper proposes a feature smoothing technique, which generates virtual data points by interpolating the input space of two randomly sampled examples. The aim is to generate virtual training data points that are close to adversarial examples. Experimental results on both MNIST and Cifar10 datasets show that the proposed method augmented with other regularization techniques are robust to adversarial attacks and obtain higher accuracy when comparing with some testing baselines. Also, the paper presents some theoretical analyses showing that label smoothing, logit squeezing, weight decay, Mixup and feature smoothing all produce small estimated variance of the decision boundary when regularizing the networks.
The paper is generally well written, and the experiments show promising results. Nevertheless, the proposed method is not very novel, and the method is not comprehensively evaluated with experiments.
Major remarks:
1.	The experiments show that feature smoothing has to combine with other regularizers in order to outperform other testing methods. In this sense the contribution of the feature smoothing along is not clear. For example, without integrating other regularizers, Mixup and feature smoothing obtain very close results for BlackBox-PGD, BlackBoxcw and Clean, as shown in Table 1. In addition, in the paper, the feature smoothing along is only validated on the MNIST (not even tested on Cifar10 in Table2). Consequently, it is difficult to evaluate the contribution of the proposed smoothing technique.
2.	Experiments are conducted on datasets MNIST and Cifar10 with small number of target classes. Empirically, it would be useful to see how it performs on more complex data set such as Cifar100 or ImageNet.
3.	The argument for why the proposed feature smoothing method works is presented in Theorem4.3 in Section 4.2, but the theorem seems to rely on the assumption that one can add data around the true decision boundary. However, how we can generate samples near the true decision boundary and how we should chose the mixing ratio to attain this goal is not clear to me in the paper. In addition, how we can sure that the adding synthetic data from one class does not collide with manifolds of other classes as suggested in AdaMixup (Guo et al., MixUp as Locally Linear Out-Of-Manifold Regularization)? This is particular relevant if the proposed feature smoothing strategy prefers to create virtual samples close to the true decision boundary.
4.	At the end of page4, the authors claim that both feature smoothing and Mixup generate new data points that are closer to the true boundary. I wonder if the authors could further justify or show that either theoretically or experimentally.
5.	The proposed method is similar to SMOTE (Chawla et al., SMOTE: Synthetic Minority Over-sampling Technique). In this sense, comparison with SMOTE would be very beneficial.
Minor remarks:
1.	In the paper Mixup, value 1 was carefully chosen as the mixing policy Alpha for Cifar10 (otherwise, underfitting can easily occur as shown in AdaMixUp), and it seems in the paper the authors used a very large value of 8 for Mixup’s Beta distribution, and I did not see the justification for that number in the paper.
2.	Typo in the second paragraph of page2: SHNV should be SVHN


The abstract paragraph should be indented 1/2 inch (3 picas) on both left and righthand margins. Use 10 point type, with a vertical spacing of 11 points. The word ABSTRACT must be centered, in small caps, and in point size 12. Two line spaces precede the abstract. The abstract must be limited to one paragraph.
Authors' names are set in boldface, and each name is placed above its corresponding address. The lead author's name is to be listed first, and the co-authors' names are set to follow. Authors sharing the same address can be on the same line.
Please pay special attention to the instructions in section 4 regarding figures, tables, acknowledgments, and references.

The authors propose a variant to convLSTMs with convolutional peephole units (as opposed to the original Hadamard product) and tied gates. The description of the model is confusing, the authors don't offer a strong justification for the proposed approach, some of the technical choices seem flawed.
It is also obviously false that removing an LSTM gate does not incur in a reduction of trainable parameters: "There were other attempts to design smaller recurrent gated models [...] based on either removing one of the gates or the activation functions [...] these models had no significant reduction in trainable parameters".
The model description is difficult to follow, but from what I can gather the model depiction is flawed. In particular, the proposed model bases on the idea to use tied gates to reduce the number of parameters. f^{(t)}, i.e., the value that governs the input, forger and output gates, is derived via a sigmoid. When the sigmoid is zero, the network will hence set all the gates to zero, ignoring the input, forgetting the memory and suppressing the output, which is hardly a behaviour one could ever want. Unfortunately, almost half of the sigmoid spectrum will lead to such behavior. This seems to be a poor modeling decision.
The nodes in Figure 1 seem to suggest that two consecutive non-linearities are applied in a row, if W_fc is clamped to zero, as reported by the authors, there is no reason to specify it, the corresponding part of the equation can be removed and the narrative simplified. In sec3, the meaning of “net input image” and “network gate image value” is unclear. Also the description of the equation is hard to follow, e.g., the square bracket notation is eventually explained only after 8 lines of text.
At the beginning of Sec4 err_1 and err_2 are defined as the difference between the predicted frame and the target frame, and vice versa. This error is then fed to the “rgcLSTM input arranger unit and to the next higher layer”. By my understanding the error of one layer is fed in the next as an input. I wonder if such error is provided also at inference time, giving a clear guidance to the network to produce the correct output exploiting privileged information. This could also explain why the training process was completed in only one epoch.
- Typos:
*Intro: More important → more importantly
* page5: ReL -> ReLu


LEARNING-BASED FREQUENCY ESTIMATION ALGORITHMS
Estimating the frequencies of elements in a data stream is a fundamental task in data analysis and machine learning. The problem is typically addressed using streaming algorithms which can process very large data using limited storage. Today's streaming algorithms, however, cannot exploit patterns in their input to improve performance. We propose a new class of algorithms that automatically learn relevant patterns in the input data and use them to improve its frequency estimates. The proposed algorithms combine the benefits of machine learning with the formal guarantees available through algorithm theory. We prove that our learning-based algorithms have lower estimation errors than their non-learning counterparts. We also evaluate our algorithms on two real-world datasets and demonstrate empirically their performance gains.

Quality/clarity:
- The problem setting description is neither formal nor intuitive which made it very hard for me to understand exactly the problem you are trying to solve. Starting with S and i: I guess S and i are both simply varying-length sequences in U.
- In general the intro should focus more on an intuitive (and/or formal) explanation of the problem setting, with some equations that explain the problem you want to work on. Right now it is too heavy on 'related work' (this is just my opinion).
Originality/Significance:
I have certainly never seen a ML-based paper on this topic. The idea of 'learning' prior information about the heavy hitters seems original.
Pros:
It seems like a creative and interesting place to use machine learning. the plots in Figure 5.2 seem promising.
Cons:
- The formalization in Paragraph 3 of the Intro is not very formal. I guess S and i are both simply varying-length sequences in U.
- In general the intro should focus more on an intuitive (and/or formal) explanation of the problem setting, with some equations that explain the problem you want to work on. Right now it is too heavy on 'related work' (this is just my opinion).
-In describing Eqn 3 there are some weird remarks, e.g. "N is the sum of all frequencies". Do you mean that N is the total number of available frequencies? i.e. should it be |D|? It's not clear to me that the sum of frequencies would be bounded if D is not discrete.
- Your F and \tilde{f} are introduced as infinite series. Maybe they should be {f1, f2,..., fN}, i.e. N queries, each of which you are trying to be estimate.
- In general, you have to introduce the notation much more carefully. Your audience should not be expected to be experts in hashing for this venue!! 'C[1,...,B]' is informal abusive notation. You should clearly state using both mathematical notation AND using sentences what each symbol means. My understanding is that that h:U->b, is a function from universe U to natural number b, where b is an element from the discrete set {1,...,B}, to be used as an index for vector C. The algorithm maintains this vector C\in N^B (ie C is a B-length vector of natural numbers). In other words, h is mapping a varying-length sequence from U to an *index* of the vector C (a.k.a: a bin). Thus C[b] denotes the b-th element/bin of C, and C[h(i)] denotes the h(i)-th element.
- Still it is unclear where 'fj' comes from. You need to state in words eg "C[b] contains the accumulation of all fj's such that h(j)=b; i.e. for each sequence j \in U, if the hash function h maps the sequence to bin b (ie $h(j)=b$), then we include the *corresponding frequency* in the sum."
- What I don't understand is how fj is dependent on h. When you say "at the end of the stream", you mean that given S, we are analyzing the frequency of a series of sequences {i_1,...,i_N}?
- Sorry, it's just confusing and I didn't really understand "Single Hash Function" from Sec 3.2 until I started typing this out.
- The term "sketch" is used in Algorithm1, like 10, before 'sketch' is defined!!
-I'm not going to trudge through the proofs, because I don't think this is self-contained (and I'm clearly not an expert in the area).
Conclusion:
Honestly, this paper is very difficult to follow. However to sum up the idea: you want to use deep learning techniques to learn some prior on the hash-estimation problem, in the form of a heavy-hitter oracle. It seems interesting and shows promising results, but the presentation has to be cleaned up for publication in a top ML venue.
******
Update after response:
The authors have provided improvements to the introduction of the problem setting, satisfying most of my complaints from before. I am raising my score accordingly, since the paper does present some novel results.


DISTRIBUTIONAL CONCAVITY REGULARIZATION FOR GANS
We propose Distributional Concavity (DC) regularization for Generative Adversarial Networks (GANs), a functional gradient-based method that promotes the entropy of the generator distribution and works against mode collapse. Our DC regularization is an easy-to-implement method that can be used in combination with the current state of the art methods like Spectral Normalization and Wasserstein GAN with gradient penalty to further improve the performance. We will not only show that our DC regularization can achieve highly competitive results on ILSVRC2012 and CIFAR datasets in terms of Inception score and Fréchet inception distance, but also provide a mathematical guarantee that our method can always increase the entropy of the generator distribution. We will also show an intimate theoretical connection between our method and the theory of optimal transport.

Summary:
This paper proposes distributional concavity regularization for GANs which encourages producing generator distributions with higher entropy. The paper motivates the proposed method as follows:
-       Using the concept of functional gradient, the paper interprets the update in the generator parameters as an update in the generator distribution
-       Given this functional gradient perspective, the paper proposes updating the generator distribution toward a target distribution which has *higher entropy and satisfies monoticity*
-       Then, the paper proves that this condition can be satisfied by ensuring that generator’s objective (L) is concave
-       Since it’s difficult to ensure concavity when parametrizing generators as deep neural networks, the paper proposes adding a simple penalty term that encourages the concavity of generator objective
Experiments confirm the validity the proposed approach. Interestingly, the paper shows that performance of multiple GAN variants can be improved with their proposed method on several image datasets

Strengths:
-   	The proposed method is very interesting and is based on sound theory
-   	Connection to optimal transport theory is also interesting
-   	In practice, the method is very simple to implement and seems to produce good results

Weaknesses:
-       Readability of the paper can be generally improved. I had to go over the paper many times to get the idea.
-       Figures should be provided with more detailed captions, which explain main result and providing context (e.g. explaining baselines).

Questions/Comments:
-       Equation (7) has typos (uses theta_old instead of theta in some places)
-       Section 4.1 (effect of monoticity) is a bit confusing. My understanding is that parameter update rule of equation (3) and (6) are equivalent, but you seem to use (6) there. Can you clarify what you do there and in general this experiment a bit more?
-       Comparing with entropy maximization method of EGAN (Dai et al, 2017) is a good idea, but I’m wondering if you can compare it on low dimensional settings (e.g. as in Fig 2). It is also not clear why increasing entropy with EGAN-VI is worse than baselines in Table 1.

Overall recommendation:
The paper is based on sound theory and provides very interesting perspective. The method seems to work in practice on a variety of experimental setting. Therefore, I recommend accepting it.


Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PACBayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.

I consider that improving the generalization capability of neural networks on small dataset is an important line of research, and the method proposed here empirically provides great results.
The proposed margin loss (Equation 1) is said to be "specially adapted for accelerating the convergence velocity of networks by [the authors]". I would like this statement to be explained better, or at least backed by empirical evidence. In the current state, I consider that the paper lacks an in-depth study of the properties of this handcrafted loss. Few is said on the benefits of having both a linear behavior for points inside the margin and a quadratic loss for far points. The impact of loss hyperparameters (r, \gamma,\mu) should be discussed thoughtfully; at some points in the paper, r and \gamma are referred as margin mean and margin variance parameters, but this interpretation is not explained. Moreover, almost nothing is said about \mu.
By considering a simplified loss function, the provided PAC-Bayes generalization bound (Theorem 1) consider solely the flat loss region [r-\gamma, r+\gamma], but shed no light on the benefit of the hinge and quadratic parts. I conceive that this might be hard to study theoretically, but the authors should at least provide a empirical study of these.
The empirical experiments show great evidence that the proposed method successfully improve generalization capability of neural networks on small datasets compared to classical methods. I appreciate the Inter/intra class variance study of Tables 2 and 3. I would like the mathematical expression of the "hinge loss" and the "soft hinge loss" models to be explicitly written (it is not clear in the text if the soft hinge uses an hyperparameter). In the same spirit of my above comments, I would like to see how each loss hyperparameters impacts the results, instead of having access solely to the parameter values selected by the validation process.
Typos and minor comments:
- Abstract: "And our ODN model also outperforms the other three loss models..." Which three loss models?
- Section 3: "Specially, define L_0 as r=\theta..." I think it should be r=0
- Section 4.1: model-s => models
- Page 7 (and elsewhere): Table. 2 => Table 2
- Please specify that "Xent" stands for cross-entropy
- Figure 3: Please use larger font sizes
- Proof of Lemma 2: Equation. 4 => Equation 4


For semantic image segmentation and lane detection, nets with a single spatial pyramid structure or encoder-decoder structure are usually exploited. Convolutional neural networks (CNNs) show great results on both high-level and low-level features representations, however, the capability has not been fully embodied for lane detection task. In especial, it's still a challenge for model-based lane detection to combine the multi-scale context with a pixel-level accuracy because of the weak visual appearance and strong prior information. In this paper, we we propose an novel network for lane detection, the three main contributions are as follows. First, we employ multiple encoder-decoders module in end-to-end ways and show the promising results for lane detection. Second, we analysis different configurations of multiple encoder-decoders nets. Third, we make our attempts to rethink the evaluation methods of lane detection for the limitation of the popular methods based on IoU.

The paper addresses the problem of pixel-wise segmentation of lanes from images taken from a vehicle-mounted camera. The proposed method uses multiple passes through encoders decoders convnets, thereby allowing extract global features to inform better local features, and vice versa. Only qualitative baseline comparisons are presented by manually comparing the output of the network to reported results of other methods in [Pan et al.2017].
It is unclear to me if the proposed multiple encoder-decoder network is a novel architecture, or a known architecture applied to a novel use case. In case of the former, more details should be given on the design of the network, how it is trained, etc. for reproducibility. The biggest problem however is the subjective manual comparison to existing methods, which the authors do in favor of a quantitative comparison using well-understand objective metrics. While they point out problems with evaluating segmentation with conventional accuracy metrics, no attempt is made to make a better objective measure. We are left to judge the results on only a few selected example frames.
It is also unclear how the method and evaluation strategy compares to methods which predict lanes as splines or other parameterized functions. E.g. see surveys on existing approaches, and discussion of different evaluation strategies, e.g. "Recent progress in road and lane detection: a survey" [Hillel et al.2014] and "Visual lane analysis and higher-order tasks: a concise review" [Shin, 2014].
Throughout the paper, various fuzzy and unclear statements are made (see detailed comments below). The paper would be in a better shape if more time is spend to improve the writing, provide more details on the method, and extend the experiments.
Pros:
+ multiple encoder-decoder stages could be beneficial for lane segmentation
Cons:
- lacking evaluation and comparison to baseline methods
- missing details on proposed network architecture, making it hard to reproduce
- unclear what colors in figures for qualitative evaluation represent: are individual lanes also distinguished?
Below are more detailed comments and questions:
* Abstract
	* "the capability has not been fully embodied for" → Fuzzy statement, I don't understand what this means.
	* "In especial" → check grammar
* Sec 1.: Introduction
	* "the local information of a lane such as sharp, edges, texture and color, can not provides distinctive features for lane detection" local edges are not distinctive for lanes? Possibly local edges alone are not sufficient, but various lanes detection approaches rely on edge extraction as features. This statement therefore seems too strong.
	* "End-to-end CNNs always give better results than systems relying on hand-crafted features.". It is not possible to say that one type of classifier categorically better than another. The 'best' classifier depends on the problem at hand, valid assumptions that can be made, and the amount of training data avaiable, among others. For instance, "How Far are We from Solving Pedestrian Detection?" [Zhang,CVPR16] demonstrates that CNNs do not always give better results than hand-crafted features for some tasks and datasets. The paper should be more careful with such strong statements.
	* "Highly hand-craft features based methods can only deal with harsh scenarios.". I don't understand, is this statement intended as an argument against hand-crafted features? Isn't it good to deal especially with harsh scenarios?
	* "but less explored on Semantic Image Segmentation due to strong prior information is needed." CNNs are extensively used for semantic image segmentation, e.g. see the well-known Cityscapes benchmark.
	* "recent methods have replaced the feature-based methods with model-based methods.". Not sure why the paper call CNNs "model-driven methods", but refer to the earlier classical methods with highly designed representations (Kalman filter, B-snakes, ...) as "feature-based methods". This seems diferent from what I typically see, where CNNs are referred to as 'data-driven methods', and the classical methods as "model-driven".
* Sec 1.2: Contributions
	* "First, reduced localization accuracy due to the weak performance of combining the local information and global information effectively and efficiently". Instead of presenting a first contribution, the paper presents a problem. Do the authors mean that they "tackle the problem of reduced localization accuracy ..." ? That would still not make this contribution very concrete though ...
	* "We make our attempts to rethink these IoU based methods." → Please argue in favor of your new method. An in-depth comparison of evaluation methods, and why some metrics fail or could be redesigned would be good. However, the paper currently fails to present a new metric, and convince that it tackles shortcomings of established metrics.
* Sec 2.: Multiple Encoder-Decoder Nets
	* Figure 2: Is this the first paper to propose this multiple encoder-decoder net? Or is the idea taken from other work, and is the novelty to apply it to this problem? If this general architecture was already proposed (for semantic segmentation?), please add citations and discuss it as related work. If this network design is completely novel, I would expect more details on how the network is constructed (e.g. dimensions of each layer, non-linear activation function used, batch normalization, strides, etc.).
	* "the following loss function:". Since it is a binary classification problem, and not a regression problem, why not use a (binary) cross entropy loss instead of a mean squared error?
* Sec 3: Experiments
	* Figure 3: What is the "Baseline" method ? Where are the references to the other works, or is the reader required to read [Pan'2017] to understand your figures?
	* Figure 3: How are the colors in these figures determined? Is this also an instance segmentation problem? From your methodology section I though only binary classification was considered. Do you do some post-processing to separate individual lanes? I find this confusing, as I thought that the task was limited to binary segmentation.
	* "Recent works evaluated ..." please cite the works you refer to.
	* "we have compared more than 500 probmaps of each level nets manually and count the accuracy of these probmaps as shown in figure 7." So if I understand correctly, instead of using an objective evaluation metric, you have reverted to manual labor to visually judge lane detection quality. This is not really a metric, and not really a solution that 'rethinks IoU based methods.' Problems of your approach is that it is unclear on what criteria results are judged, your evaluation is not objectively reproducible by others, and does not scale well for novel future evaluations. Why is this even needed? E.g. why not use some chamfer distance or Gaussian smoothing of the edge map if you want to evaluate near coverage instead of hard boundaries? Or, fit a function through the boundary, and evaluate distance (in meters) to true lane. I find the proper discussion and motivation for manual evaluation over objective metric evaluation lacking.
	* Figure 6: What are the Ground Truth images of each row ? E.g. in the fourth row from the top, should the right-most yellow lane be present or not? As it stands, I can't interpret the columns and see which x times is visually 'better'.
* Sec 3.4:
	* "To improve ability of the network, we propose a small quantity of channel to reduce overfitting by considering inter-dependencies among channels." To improve relative to what? Where are the results comparing large amounts vs small amount of channels? Note that Figure 8 is not referred to in the text, and confusingly compares "18 layers" to "1 layers". Do you mean channels instead of layers? And, how many channels were to obtained the results in the preceding sections?


ABSTRACT k-Nearest Neighbors is one of the most fundamental but effective classification models. In this paper, we propose two families of models built on a sequence to sequence model and a memory network model to mimic the k-Nearest Neighbors model, which generate a sequence of labels, a sequence of out-of-sample feature vectors and a final label for classification, and thus they could also function as oversamplers. We also propose 'out-of-core' versions of our models which assume that only a small portion of data can be loaded into memory. Computational experiments show that our models outperform k-Nearest Neighbors, a feedforward neural network and a memory network, due to the fact that our models must produce additional output and not just the label. As an oversampler on imbalanced datasets, the sequence to sequence kNN model often outperforms Synthetic Minority Over-sampling Technique and Adaptive Synthetic Sampling.

This work uses sequence-to-sequence and memory network neural nets to learn a
network that not only predicts a label, but also predicts nearest neighbors and
their label.  The intuition is that by training on a related but harder task,
the network is forced to learn not just about sampled points but about the
behavior in regions around the actual training examples.
Their work shows that imposing these additional requirements on the model does
result in better performance on unseen data, where only the label of the unseen
data point is required as output.  They show that learning more about the
global {feature,label} distribution improves F1 scores, and suggest that their
methods can be used as an example generator for datasets with class imbalance.
The writing was clear.  I had only 1 misunderstanding that cause me trouble,
namely the first sentence of "Classification", where I might put the word
'benchmark' up front rather than at the end of this somewhat long sentence.
By the time 'knn benchmark' appeared some paragraphs later, I realized I
had missed something, and had to backtrack.
Figs 1--3 are only viewable on screen with fairly extreme magnification. I
found different viewers varied in legibility at these extreme zoom settings.
However, when expanded so y-axis numbers could be deciphered, it turned out
that actual improvements in F1 score were rather modest.
For 4 datasets, their F1 scores of their best method, V2VSLS, improved by ~ 1.5
to 6%.  They found that their methods were less affected by out-of-core
computation than the benchmark kNN F1 scores.
I did have some questions about the problem formulation.  I did not really
understand why the models wanted to reproduce the exact ordering of the
nearest neighbors, other than this is easy to formulate.  This makes sense
for n.n. label sequences, where further points might be in some neighboring
class.  But for predicting the feature vectors of n.n., it seems that the
exact order is not robust, in the sense very small distance changes can cause
abrupt shifts in the target nearest-neghbor ordering.  Is there some way
for me to understand why this does not pose a problem?  Or is there perhaps
a way to make the loss function a bit less dependent on the precise order
of the generated feature vector sequence?
In the computational experiments, with the choice of datasets, it was often
hard for me to judge how much different aspects of their 4 network structures
were really being excercised.  The main issue is the all problems used only
2--3 classes.  I could not guess what fraction of data points had actual label
changes within the 5 n.n..  For many datasets, "same class" might be a pretty
good predictor of nearest neighbor label.
Alternatively, one can consider the other extreme, of very many classes.
Does it still make sense to try to predict the order of nearest neighbor labels in
such a setting?
The authors provide evidence of some performance improvement, but I would
encourage them to provide some intuition about what the networks are actually
learning.  Some of this can be done with their existing data.
For example, on average, what are the distances from predicted feature vectors
to the actual nn feature vectors?  If the feature sequence is typically badly
predicted, then this might allow the authors to propose that the network is
*actually* learning some simpler features of the the distributions underlying
an actual {feature,label} sequence.  This might allow simplified training losses,
based, on things like direction and distance to same-label cluster center,
direction+distance to average same-class nn.s, direction and distance to
closest differently labeled cluster, etc.  Or does their data suggest that
their models are actually learning the precise nearest neighbor ordering?
Such considerations might be able to improve the OOC training, since
"global" aspects of the distribution features (like "cluster center")
remain approx. valid as training batches changes.
The other question I had was with the oversampling proposition.  The extent
of class imbalance in the datasets is not described.  Perhaps it belongs in
Table 1.  Their approach seems a lot of work for modest gains usually available
with oversampling.  Can the authors provide any guideline for how many members a
minority class should have before using their sequence-to-sequence technique?
Pros: they improve generalization to unseen data.
Cons: their models are considerably more complex, and they do not analyze their
data in enough detail to suggest whether their complexity is necessary, or perhaps
could be reduced.  Figures are too small (many unreadable in printed copy).
Datasets have very few classes and the extent to which nearest neighbors are
of different class not reported.


ON THE MINIMAL SUPERVISION FOR TRAINING ANY BINARY CLASSIFIER FROM ONLY UNLABELED DATA
Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question-what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.

This paper studies the weak supervision setting of learning a general binary classifier from two unlabeled (U) datasets with known class balances. The authors establish that this is possible by constructing an unbiased estimator, analyze its convergence theoretically, and then run experiments using modern image classification models.
Pros:
- This work demonstrates, theoretically and empirically, a simple way to train generic models using only the known class balances of several sets of unlabeled data (having the same conditional distributions p(x|y))---a very interesting configuration of weak supervision, an increasingly popular and important area
- The treatment is thorough, proceeding from establishing the minimum number of U datasets, constructing the estimator, analyzing convergence, and implementing thorough experiments
Cons:
- This is a crowded area (as covered in their related work section). As they cite, (Quadrianto et al., 2009) proposed this setting and considered linear models for k-wise classification.  Moreover, the two U datasets with known class balances can equivalently be viewed as two weak / noisy label sources with known accuracies.  Thus this work connects to many areas- both in noisy learning, as they cite heavily, but also in methods (in e.g. crowdsourcing and multi-source weak supervision) where several sources label unlabeled datasets with unknown accuracies (which are often estimated in an unsupervised fashion).
- The overall clarity of the paper's writing could be improved. For example, the introduction and related work sections take up a large portion of the paper, but are very dense and heavy with jargon that is not internally defined upfront; for example "risk rewrite" is introduced in paragraph 2 with no internal definition and then used subsequently throughout the paper (this defn would be simple enough to give: in the context of this paper, "risk rewrite" means a linear combination of the class-conditional losses; or more generally, the expected loss w.r.t. distribution over classes...).  Also intuition could be briefly given about the theorem proof strategies.
- The difference between the two class distributions over the U datasets seems like an important quantity (akin, in e.g. weak supervision / crowd source modeling papers, to quantity of how bounded away from random noise the labelers are). This is treated empirically, but would be stronger to have this show up in the theory somewhere.
- Other prior work here has handled k classes with k U sets; could have extended to cover this setting too, since seems natural
Overall take: This learning from label proportions setting has been covered before, but this paper presents it in an overall clean and general way, testing it empirically on modern models and datasets, which is an interesting contribution.
Other minor points:
- The argument for / distinction between using eqns. (3) and (4) seems a bit ad hoc / informal ("we argue that...").  This is an important point...
- Theorem 1 proof seems fine, but some intuition in the main body would be nice.
- What does "classification calibrated" mean?
- Saying that three U sets are needed, where this includes the test set, seems a bit non-standard?  Also I'm confused- isn't a labeled test set used?  So what is this third U set for?
- The labels l_+ and l_- in Defn. 3 seem to imply that the two U sets are positive vs. negative; but this is not the case, correct…?
- Stating both Lemma 5 and Thm 6 seems unnecessary
- In Fig. 2, seems like could have trained for longer and perhaps some of the losses would have continued decreasing?  In particular, small PN?  Also, a table of the final test set accuracies would have been very helpful.
- More detail on experimental protocol would be helpful: what kind of hyperparameter tuning was done? repeated runs averaging?  It seems odd, for example in Fig. 3, that the green lines are so different in (a) vs. (c), and not in the way that one would expect given the decrease in theta


Accurate rent prediction in real estate investment can help in generating capital gains and guaranty a financial success. In this paper, we carry out a comprehensive analysis and study of eleven machine learning algorithms for rent prediction, including Linear Regression, Multilayer Perceptron, Random Forest, KNN, ML-KNN, Locally Weighted Learning, SMO, SVM, J48, lazy Decision Tree (i.e., lazy DT), and KStar algorithms. Our contribution in this paper is twofold: (1) We present a comprehensive analysis of internal and external attributes of a real-estate housing dataset and their correlation with rental prices. (2) We use rental prediction as a platform to study and compare the performance of eager vs. lazy machine learning methods using myriad of ML algorithms. We train our rent prediction models using a Zillow data set of 4K real estate properties in Virginia State of the US, including three house types of single-family, townhouse, and condo. Each data instance in the dataset has 21 internal attributes (e.g., area space, price, number of bed/bath, rent, school rating, so forth). In addition to Zillow data, external attributes like walk/transit score, and crime rate are collected from online data sources. A subset of the collected features -determined by the PCA techniqueare selected to tune the parameters of the prediction models. We employ a hierarchical clustering approach to cluster the data based on two factors of house type, and average rent estimate of zip codes. We evaluate and compare the efficacy of the tuned prediction models based on two metrics of R-squared and Mean Absolute Error, applied on unseen data. Based on our study, lazy models like KStar lead to higher accuracy and lower prediction error compared to eager methods like J48 and LR. However, it is not necessarily found to be an overarching conclusion drawn from the comparison between all the lazy and eager methods in this work.

The paper investigates different machine learning approaches to model and
predict the return on property investments, in particular with respect to eager
and lazy learning techniques. The authors evaluate those different techniques on
a dataset of properties in Virginia. They conclude that lazy techniques provide
better performance than eager ones.
The paper is a purely empirical study that does not introduce any novel
machine learning or evaluation techniques. The authors use the off-the-shelf
WEKA toolbox. The results are not clear, given that only a single data set was
used to evaluate the different approaches, and general recommendations cannot be
made.
The paper is not well written and the descriptions do not convey what the
authors have done very well. An example of this is Figure 3, which purports to
show the average rent (or rent distribution?) for different housing types. There
are multiple categories in there that are not valid housing types ("Make Me
Move") and the rents shown are incorrect (e.g. more than a million for a town
house). It is also unclear why the average rent for a single family house is
approximately 4 times as much as for a town house.
The problems with Figure 3 are exemplary of the paper; the other issues are too
numerous to list.
In summary, this paper should be rejected.


Predicting the time to the next event is an important task in various domains. However, due to censoring and irregularly sampled sequences, time-to-event prediction has resulted in limited success only for particular tasks, architectures and data. Using recent advances in probabilistic programming and density networks, we make the case for a generalized parametric survival approach, sequentially predicting a distribution over the time to the next event. Unlike previous work, the proposed method can use asynchronously sampled features for censored, discrete, and multivariate data. Furthermore, it achieves good performance and near perfect calibration for probabilistic predictions without using rigid network-architectures, multitask approaches, complex learning schemes or non-trivial adaptations of coxmodels. We firmly establish that this can be achieved in the standard neural network framework by simply switching out the output layer and loss function.

The authors propose a parametric framework (HazardNet) for survival analysis with deep learning where they mainly focus on the discrete-time case. The framework allows different popular architectures to learn the representation of the past events with different explicit features. Then, it considers a bunch of parametric families and their mixtures for the distribution of inter-event time. Experiments include a comprehensive comparison between HazardNet and different binary classifiers trained separately at each target time duration.
Overall, the paper is well-written and easy to follow. It seeks to build a strong baseline for the deep survival analysis, which is an hot ongoing research topic recently in literature. However, there are a few weaknesses that should be addressed.
1. In the beginning, the paper motivates the mixtures of distributions from MDN. Because most existing work focuses on the formulation of the intensity function, it is very interesting to approach the problem from the cumulative intensity function instead. Originally, it looks like the paper seeks to formulate a general parametric form based on MDN. However, it is disappointing that in the experiments, it still only considers a few classic parametric distributions. There is lack of solid technical connection between Sec 3.1, 3.2 and Sec 4.
2. The discretization discussion of Sec 3.4 is not clear. Normally, the major motivation for discretization is application-driven, say, in hospital, the doctor regularly triggers the inspection event. However, how to optimally choose a bin-size and how to aggregate the multiple events within each bin is still not clear, which is not sufficiently discussed in the paper. Why is taking the summation of the events in a bin a proper way of aggregation? What if we have highly skewed bins?
3. Although the comparison and experimental setting in Figure 4 is comprehensive, the paper misses a very related work "Deep Recurrent Survival Analysis, https://arxiv.org/abs/1809.02403", which also considers the discrete-time version of survival analysis. Only comparing with the binary classifiers is not quite convincing without referring to other survival analysis work.
4. Finally, the authors state that existing temporal point process work "have little meaning without taking into account censored data". However, if inspecting the loss function of these work closely, we can see there is a survival term exactly the same as the log-cumulative hazard in Equation 3 that handles the censored case.
5. A typo on the bottom of page 3, should be p(t) = F(t + 1) - F(t)


Temporal Difference learning with function approximation has been widely used recently and has led to several successful results. However, compared with the original tabular-based methods, one major drawback of temporal difference learning with neural networks and other function approximators is that they tend to over-generalize across temporally successive states, resulting in slow convergence and even instability. In this work, we propose a novel TD learning method, Hadamard product Regularized TD (HR-TD), to reduce over-generalization and thus leads to faster convergence. This approach can be easily applied to both linear and nonlinear function approximators. HR-TD is evaluated on several linear and nonlinear benchmark domains, where we show improvement in learning behavior and performance.

This paper introduces a variation on temporal difference learning for the function approximation case that attempts to resolve the issue of over-generalization across temporally-successive states. The new approach is applied to both linear and non-linear function approximation, and for prediction and control problems. The algorithmic contribution is demonstrated with a suite of experiments in classic benchmark control domains (Mountain Car and Acrobot), and in Pong.
This paper should be rejected because (1) the algorithm is not well justified either by theory or practice, (2) the paper never clearly demonstrates the existence of problem they are trying to solve (nor differentiates it from the usual problem of generalizing well), (3) the experiments are difficult to understand, missing many details, and generally do not support a significant contribution, and (4) the paper is imprecise and unpolished.
Main argument
The paper does not do a great job of demonstrating that the problem it is trying to solve is a real thing. There is no experiment in this paper that clearly shows how this temporal generalization problem is different from the need to generalize well with function approximation. The paper points to references to establish the existence of the problem, but for example the Durugkar and Stone paper is a workshop paper and the conference version of that paper was rejected from ICLR 2018 and the reviewers highlighted serious issues with the paper—that is not work to build upon. Further the paper under review here claims this problem is most pressing in the non-linear case, but the analysis in section 4.1 is for the linear case.
The resultant algorithm does not seem well justified, and has a different fixed point than TD, but there is no discussion of this other than section 4.4, which does not make clear statements about the correctness of the algorithm or what it converges to. Can you provide a proof or any kind of evidence that the proposed approach is sound, or how it’s fixed point relates to TD?
The experiments do not provide convincing evidence of the correctness of the proposed approach or its utility compared to existing approaches. There are so many missing details it is difficult to draw many conclusions:
1) What was the policy used in exp1 for policy evaluation in MC?
2) Why Fourier basis features?
3) In MC with DQN how did you adjust the parameters and architecture for the MC task?
4) Was the reward in MC and Acrobot -1 per step or something else
5) How did you tune the parameters in the MC and Acrobot experiments?
6) Why so few runs in MC, none of the results presented are significant?
7) Why is the performance so bad in MC?
8) Did you evaluate online learning or do tests with the greedy policy?
9) How did you initialize the value functions and weights?
10) Why did you use experience replay for the linear experiments?
11) IN MC and Acrobot why only a one layer MLP?
Ignoring all that, the results are not convincing. Most of the results in the paper are not statistically significant. The policy evaluation results in MC show little difference to regular TD. The Pong results show DQN is actually better. This makes the reader wonder if the result with DQN on MC and Acrobot are only worse because you did not properly tune DQN for those domains, whereas the default DQN architecture is well tuned for Atari and that is why you method is competitive in the smaller domains.
The differences in the “average change in value plots” are very small if the rewards are -1 per step. Can you provide some context to understand the significance of this difference? In the last experiment linear FA and MC, the step-size is set equal for all methods—this is not a valid comparison. Your method may just work better with alpha = 0.1.
The paper has many imprecise parts, here are a few:
1) The definition of the value function would be approximate not equals unless you specify some properties of the function approximation architecture. Same for the Bellman equation
2) equation 1 of section 2.1 is neither an algorithm or a loss function
3) TD does not minimize the squared TD. Saying that is the objective function of TD learning in not true
4) end of section 2.1 says “It is computed as” but the following equation just gives a form for the partial derivative
5) equation 2, x is not bounded
6) You state TC-loss has an unclear solution property, I don’t know what that means and I don’t think your approach is well justified either
7) Section 4.1 assumes linear FA, but its implied up until paragraph 2 that it has not assumed linear
8) treatment of n_t in alg differs from appendix (t is no time episode number)
9) Your method has a n_t parameter that is adapted according to a schedule seemingly giving it an unfair advantage over DQN.
10) Over-claim not supported by the results: “we see that HR-TD is able to find a representation that is better at keeping the target value separate than TC is “. The results do not show this.
11) Section 4.4 does not seem to go anywhere or produce and tangible conclusions
Things to improve the paper that did not impact the score:
0) It’s hard to follow how the prox operator is used in the development of the alg, this could use some higher level explaination
1) Intro p2 is about bootstrapping, use that term and remove the equations
2) Its not clear why you are talking about stochastic vs deterministic in P3
3) Perhaps you should compare against a MC method in the experiments to demonstrate the problem with TD methods and generalization
4) Section 2: “can often be a regularization term” >> can or must be?
5) update law is a odd term
6)” tends to alleviate” >> odd phrase
7) section 4 should come before section 3
8) Alg 1 in not helpful because it just references an equation
9) section 4.4 is very confusing, I cannot follow the logic of the statements
10) Q learning >> Q-learning
11) Not sure what you mean with the last sentence of p2 section 5
12) where are the results for Acrobot linear function approximation
13) appendix Q-learning with linear FA is not DQN (table 2)


The paper, interested in unsupervised feature selection, aims to retain the features best accounting for the local patterns in the data. The proposed approach, called Locally Linear Unsupervised Feature Selection (LLUFS), relies on a dimensionality reduction method to characterize such patterns; each feature is thereafter assessed according to its compliance w.r.t. the local patterns, taking inspiration from Locally Linear Embedding (Roweis and Saul, 2000). The experimental validation of the approach on the scikit-feature benchmark suite demonstrates its effectiveness compared to the state of the art.

Summary: The paper proposes the LLUFS method for feature selection. The idea is to first apply a dimensionality reduction method on the input data X to find a low-dimensional representation Z. Next, each point in Z is represented by a linear combination of its nearest neighbors by finding a matrix W which minimizes || Z  - WZ||. Finally, these weights are used to asses the distortion of every feature in X by considering the reconstruction loss in the original space.
Comments: There are multiple shortcomings in the motivation of the approach. First, the result of the dimensionality reduction drastically depend on the method used. It is well known that every DR method focuses on preserving certain properties of the data. For instance, PCA preserves the global structure while t-SNE works locally, maximizing the recall [1]. The choice of the DR method should justify the underlying assumption of the approach. I expect that the results of the experiments to change drastically by changing the DR method.
Second, the LLE method is based on the assumption that if the high-dimensional data is locally linear, it can be projected on a low-dimensional embedding which is also locally linear. Transitioning from a locally linear high-dimensional data to a lower dimension makes sense because there exists higher degree of freedom in the higher dimension. However, making this assumption in the opposite direction is not very intuitive. Why would the features that do not conform to the local linearity of the low-dimensional structure (which itself is obtained via a non-linear mapping) are insignificant?
Finally, there are no theoretical guarantees on the performance of the method. Is there any guarantee that, e.g. given one noisy feature in high dimension, the method will find that feature, etc.?
Minor: what is the complexity of the method compared to the competing methods? What is the runtime? Is this a practical approach on large datasets?
Overall, I do not agree with the assumptions of the paper nor convinced with the experimental study. Therefore, I vote for reject.
[1] Venna et al. "Information retrieval perspective to nonlinear dimensionality reduction for data visualization." Journal of Machine Learning Research 11, no. Feb (2010): 451-490.


DHER: HINDSIGHT EXPERIENCE REPLAY FOR DYNAMIC GOALS
Dealing with sparse rewards is one of the most important challenges in reinforcement learning (RL), especially when a goal is dynamic (e.g., to grasp a moving object). Hindsight experience replay (HER) has been shown an effective solution to handling sparse rewards with fixed goals. However, it does not account for dynamic goals in its vanilla form and, as a result, even degrades the performance of existing off-policy RL algorithms when the goal is changing over time. In this paper, we present Dynamic Hindsight Experience Replay (DHER), a novel approach for tasks with dynamic goals in the presence of sparse rewards. DHER automatically assembles successful experiences from two relevant failures and can be used to enhance an arbitrary off-policy RL algorithm when the tasks' goals are dynamic. We evaluate DHER on tasks of robotic manipulation and moving object tracking, and transfer the polices from simulation to physical robots. Extensive comparison and ablation studies demonstrate the superiority of our approach, showing that DHER is a crucial ingredient to enable RL to solve tasks with dynamic goals in manipulation and grid world domains.

In this paper, the authors extend the HER framework to deal with dynamical goals, i.e. goals that change over time.
In order to do so, they first need to learn a model of the dynamics of the goal, and then to select in the replay buffer experience reaching the expected value of the goal at the expected time. Empirical results are based on three (or four, see the appendix) experiments with a Mujoco UR10 simulated environment, and one experiment is successfully transfered to a real robot.
Overall, the addressed problem is relevant (the question being how can you efficiently replay experience when the goal is dynamical?), the idea is original and the approach looks sound, but seems to suffer from a fundamental flaw (see below).
Despite some merits, the paper mainly suffers from the fact that the implementation of the approach described above is not explained clearly at all.
Among other things, after reading the paper twice, it is still unclear to me:
- how the agent learns of the goal motion (what substrate for such learning, what architecture, how many repetitions of the goal trajectory, how accurate is the learned model...)
- how the output of this model is taken as input to infer the desired values of the goal in the future: shall the agent address the goal at the next time step or later in time, how does it search in practice in its replay buffer, etc.
These unclarities are partly due to unsufficient structuring of the "methodology" section of the paper, but also to unsufficient mastery of scientific english. At many points it is not easy to get what the authors mean, and the paper would definitely benefit from the help of an experienced scientific writer.
Note that Figure 1 helps getting the overall idea, but another Figure showing an architecture diagram with the main model variables would help further.
In Figures 3a and 5, we can see that performance decreases. The explanation of the authors just before 4.3.1 seem to imply that there is a fundamental flaw in the algorithm, as this may happen with any other experiment. This is an important weakness of the approach.
To me, Section 4.5 about transfer to a real robot does not bring much, as the authors did nothing specific to favor this transfer. They just tried and it happens that it works, but I would like to see a discussion why it works, or that the authors show me with an ablation study that if they change something in their approach, it does not work any more.
In Section 4.6, the fact that DHER can outperform HER+ is weird: how can a learn model do better that a model given by hand, unless that given model is wrong? This needs further investigation and discussion.
In more details, a few further remarks:
In related work, twice: you should not replace an accurate enumeration of papers with "and so on".
p3: In contrary, => By contrast,
which is the same to => same as
compare the above with the static goals => please rephrase
In Algorithm 1, line 26: this is not the algorithm A that you optimize, this is its critic network.
line 15: you search for a trajectory that matches the desired goal. Do you take the first that matches? Do you take all that match, and select the "best" one? If yes, what is the criterion for being the best?
p5: we find such two failed => two such failed
that borrows from the Ej => please rephrase
we assign certain rules to the goals so that they accordingly move => very unclear. What rules? Specified how? Please give a formal description.
For defining the reward, you use s_{t+1} and g_{t+1}, why not s_t and g_t?
p6: the same cell as the food at a certain time step. Which time step? How do you choose?
The caption of Fig. 6 needs to be improved to be contratsed with Fig. 7.
p8: the performance of DQN and DHER is closed => close?
DHER quickly acheive(s)
Because the law...environment. => This is not a sentence.
Mentioning in the appendix a further experiment (dy-sliding) which is not described in the paper is of little use.


QUASI-HYPERBOLIC MOMENTUM AND ADAM FOR DEEP LEARNING
Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-theart result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

Edit: Following response, I have updated my score from 6 to 7.
I completed this review as an emergency reviewer - meaning that I had little time to complete the review. I did not have time to cover all of the material in the lengthy appendix but hope that I explored the parts most relevant to my comments below.
Paper summary: The paper introduces QHM, a simple variant of classical momentum which takes a weighted average of the momentum and gradient update. The authors comprehensively analyze the relationships between QHM and other momentum based optimization schemes. The authors present an empirical evaluation of QHM and QHAdam showing comparable performance with existing approaches.
Detailed comments:
I'll use CM to denote classical momentum, referred to as "momentum" in the paper.
1) In the introduction, you reference gradient variance reduction as a motivation for QHM. But in Section 3 you defer readers to the appendix for the motivation of QHM. I think that the main paper should include a brief explanation of this motivation.
2) The proposed QHM looks quite similar to a special case of Aggregated Momentum [1]. It seems that the key difference is with the use of damping but I suspect that this can be largely eliminated by using different learning rates for each velocity (as in Section 4 of [1]) and/or adopting damping in AggMo. In fact, Section 4.1 in your paper recovers Nesterov momentum in a very similar way. More generally, could one think of AggMo as a generalization of QHM? It averages plain SGD and several momentum steps on different time scales.
3) I thought that some of the surprising relations to other momentum based optimizers was the most interesting part of the paper. However, I found the presentation a little difficult. There are many algorithms presented but none are explored fully in the main paper. I had to flick between the main paper and appendix to uncover the information I wanted most from the paper.
Moreover, I found some of the arguments in the appendix a little tough to follow. For example, with AccSGD you should specify that epsilon is a constant typically chosen to be 0.7.  When the correspondence to QHM is presented it is not obvious that QHM -> AccSGD but not the other way around. I would suggest that you present a few algorithms in greater detail, and list the other algorithms you explore at the end of Section 4 with pointers to the appendix.
4) I am not sure that the QHAdam algorithm adds much to the paper. It is not explored theoretically and I found the empirical analysis fairly limited.
5) In general, the empirical results support QHM as an improvement on SGD/NAG. But I have some (fairly minor) concerns.
   a) For Figure 1, it looks like QHM beats QHAdam on MLP-EMNIST. Why not show these on the same plot? This goes back to my point 4 - it does not look like QHAdam improves on QHM and so I am not sure why it is included. The idea of averaging gradients and momentum is general - why explore QHAdam in particular?
   b) For Figure 2, while I certainly appreciate the inclusion of error bars, they suggest that the performance of all methods are very similar. In Table 3, QH and the baselines are often not just within a standard deviation of eachother but also have very close means (relatively).
6) I feel that some of the claims made in the paper are a little strong. E.g. "our algorithms lead to significantly improved training in a variety of settings". I felt that the evidence for this was lacking.
Overall, I felt that the paper offered many interesting results but clarity could be improved. I have some questions about the empirical results but felt that the overall story was strong. I hope that the issues I presented above can be easily addressed by the authors.
Minor comments:
- I thought the use of bold text in the introduction was unnecessary
- Some summary of the less common tasks in Table 2 should be given in the main paper
Clarity: I found the paper quite difficult to follow in places and found myself bouncing around the appendix frequently. While the writing is good I think that some light restructuring would improve the flow.
Significance: The paper presents a simple tweak to classical momentum but takes care to identify its relation to existing algorithms. The empirical results are not overwhelming but at least show QHM as competitive with CM on tasks and architecture where SGD is typically dominant.
Originality: To my knowledge, the paper presents original findings and places itself well amongst existing work.
References:
[1] Lucas et al. "Aggregated Momentum: Stability Through Passive Damping" https://arxiv.org/pdf/1804.00325.pdf


We introduce and demonstrate the variational autoencoder (VAE) for probabilistic non-negative matrix factorisation (PAE-NMF). We design a network which can perform non-negative matrix factorisation (NMF) and add in aspects of a VAE to make the coefficients of the latent space probabilistic. By restricting the weights in the final layer of the network to be non-negative and using the non-negative Weibull distribution we produce a probabilistic form of NMF which allows us to generate new data and find a probability distribution that effectively links the latent and input variables. We demonstrate the effectiveness of PAE-NMF on three heterogeneous datasets: images, financial time series and genomic.

TITLE
A VARIATIONAL AUTOENCODER FOR PROBABILISTIC NON-NEGATIVE MATRIX FACTORISATION
REVIEW SUMMARY
Well written, interesting new idea, modest technical contribution, limited demonstration.
PAPER SUMMARY
The paper presents an approach to NMF within a variational autoencoder framework. It uses a Weibull distribution in the latent space.
QUALITY
The work appears technically sound except for minor typos.
CLARITY
Overall the paper is a pleasure to read. Only the presentation of the standard vae could be more clear.
ORIGINALITY
The method is (to my knowledge) novel.
SIGNIFICANCE
I think this paper is a significant contribution. I feel I have learned something from reading it, and am motivated to try out this approach. I believe there should be a wide general interest. The technical contribution is perhaps somewhat modest, as the paper fairly straightforwardly includes non-negativity in a vae setting, but I think this is a good idea. The demonstration of the algorithm is also quite limited - I would have enjoyed seeing this applied to some more reaslistic, practical problems, where perhaps the quantification of uncertaincy (which is one of the main benefits of a vae-based nmf) would come more directly into play.
FURTHER COMMENTS
page 3
The presentation of the VAE objective is a bit oblique. The statement "they require a different objectiv function" is not wrong, but also not very precise. The equality in eq. (2) is incorrect (I assume this is meant to be a stochastic approximation, i.e. the expectation over q approximated by sampling?)
"with \hat v  the reconstructed vector" Not clear. I assume \hat v is reconstructed from a sample from q given v ?
There is a typo in eq. 3. The first factor in the second to last term should be (lambda_1/\lambda_2)^(k_2)