In representational lifelong learning an agent aims to learn to solve novel tasks while updating its representation in light of previous tasks. Under the assumption that future tasks are 'related' to previous tasks, representations should be learned in such a way that they capture the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of a new task. We develop a framework for lifelong learning in deep neural networks that is based on generalization bounds, developed within the PAC-Bayes framework. Learning takes place through the construction of a distribution over networks based on the tasks seen so far, and its utilization for learning a new task. Thus, prior knowledge is incorporated through setting a history-dependent prior for novel tasks. We develop a gradient-based algorithm implementing these ideas, based on minimizing an objective function motivated by generalization bounds, and demonstrate its effectiveness through numerical examples.

The paper considers multi-task setting of machine learning. The first contribution of the paper is a novel PAC-Bayesian risk bound. This risk bound serves as an objective function for multi-task machine learning. A second contribution is an algorithm, called LAP, for minimizing a simplified version of this objective function. LAP algorithm uses several training tasks to learn a prior distribution P over hypothesis space. This prior distribution P is then used to find a posterior distribution Q that minimizes the same objective function over the test task. The third contribution is an empirical evaluation of LAP over toy dataset of two clusters and over MNIST.
While the paper has the title of "life-long learning", the authors admit that all experiments are in multi-task setting, where
the training is done over all tasks simultaneously. The novel risk bound and LAP algorithm can definitely be applied to life-long setting, where training tasks are available sequentially. But since there is no empirical evaluation in this setting, I suggest to adjust the title of the paper.

The novel risk bound of the paper is an extension of the bound from [Pentina & Lampert, ICML 2014]. The extension seems to be quite significant. Unlike the bound of [Pentina & Lampert, ICML 2014], the new bound allows to re-use many different PAC-Bayesian complexity terms that were published previously.
I liked risk bound and optimization sections of the paper. But I was less convinced by the empirical experiments. Since
the paper improves the risk bound of [Pentina & Lampert, ICML 2014], I expected to see an empirical comparison of LAP and optimization  algorithm from the latter paper. To make such comparison fair, both optimization algorithms should use the same base algorithm, e.g. ridge regression, as in [Pentina & Lampert, ICML 2014]. Also I suggest to use the datasets from the latter paper.
The experiment with multi-task learning over MNIST dataset looks interesting, but it is still a toy experiment. This experiment  will be more convincing with more sophisticated datasets (CIFAR-10, ImageNet) and architectures (e.g. Inception-V4, ResNet).
Minor remarks:
Section 6, line 4: "Combing" -> "Combining"
Page 14, first equation: There should be "=" before the second expectation.


Published as a conference paper at ICLR 2018 LEARNING TO CLUSTER IN ORDER TO TRANSFER ACROSS DOMAINS AND TASKS
This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster. The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning. We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not (pairwise semantic similarity). This similarity is category-agnostic and can be learned from data in the source domain using a similarity network. We then present two novel approaches for performing transfer learning using this similarity function. First, for unsupervised domain adaptation, we design a new loss function to regularize classification with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs. Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network. Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches. Using this method, we first show state of the art results for the challenging cross-task problem, applied on Omniglot and ImageNet. Our results show that we can reconstruct semantic clusters with high accuracy. We then evaluate the performance of cross-domain transfer using images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both datasets. Our approach doesn't explicitly deal with domain discrepancy. If we combine with a domain adaptation loss, it shows further improvement.

The authors propose a method for performing transfer learning and domain adaptation via a clustering approach. The primary contribution is the introduction of a Learnable Clustering Objective (LCO) that is trained on an auxiliary set of labeled data to correctly identify whether pairs of data belong to the same class. Once the LCO is trained, it is applied to the unlabeled target data and effectively serves to provide "soft labels" for whether or not pairs of target data belong to the same class. A separate model can then be trained to assign target data to clusters while satisfying these soft labels, thereby ensuring that clusters are made up of similar data points.
The proposed LCO is novel and seems sound, serving as a way to transfer the general knowledge of what a cluster is without requiring advance knowledge of the specific clusters of interest. The authors also demonstrate a variety of extensions, such as how to handle the case when the number of target categories is unknown, as well as how the model can make use of labeled source data in the setting where the source and target share the same task.
The way the method is presented is quite confusing, and required many more reads than normal to understand exactly what is going on. To point out one such problem point, Section 4 introduces f, a network that classifies each data instance into one of k clusters. However, f seems to be mentioned only in a few times by name, despite seeming like a crucial part of the method. Explaining how f is used to construct the CCN could help in clarifying exactly what role f plays in the final model. Likewise, the introduction of G during the explanation of the LCO is rather abrupt, and the intuition of what purpose G serves and why it must be learned from data is unclear. Additionally, because G is introduced alongside the LCO, I was initially misled into understanding was that G was optimized to minimize the LCO. Further text explaining intuitively what G accomplishes (soft labels transferred from the auxiliary dataset to the target dataset) and perhaps a general diagram of what portions of the model are trained on what datasets (G is trained on A, CCN is trained on T and optionally S') would serve the method section greatly and provide a better overview of how the model works.
The experimental evaluation is very thorough, spanning a variety of tasks and settings. Strong results in multiple settings indicate that the proposed method is effective and generalizable. Further details are provided in a very comprehensive appendix, which provides a mix of discussion and analysis of the provided results. It would be nice to see some examples of the types of predictions and mistakes the model makes to further develop an intuition for how the model works. I'm also curious how well the model works if, you do not make use of the labeled source data in the cross-domain setting, thereby mimicking the cross-task setup.
At times, the experimental details are a little unclear. Consistent use of the A, T, and S' dataset abbreviations would help. Also, the results section seems to switch off between calling the method CCN and LCO interchangeably. Finally, a few of the experimental settings differ from their baselines in nontrivial ways. For the Office experiment, the LCO appears to be trained on ImageNet data. While this seems similar in nature to initializing from a network pre-trained on ImageNet, it's worth noting that this requires one to have the entire ImageNet dataset on hand when training such a model, as opposed to other baselines which merely initialize weights and then fine-tune exclusively on the Office data. Similarly, the evaluation on SVHN-MNIST makes use of auxiliary Omniglot data, which makes the results hard to compare to the existing literature, since they generally do not use additional training data in this setting. In addition to the existing comparison, perhaps the authors can also validate a variant in which the auxiliary data is also drawn from the source so as to serve as a more direct comparison to the existing literature.
Overall, the paper seems to have both a novel contribution and strong technical merit. However, the presentation of the method is lacking, and makes it unnecessarily difficult to understand how the model is composed of its parts and how it is trained. I think a more careful presentation of the intuition behind the method and more consistent use of notation would greatly improve the quality of this submission.
=========================
Update after author rebuttal:
=========================
I have read the author's response and have looked at the changes to the manuscript. I am satisfied with the improvements to the paper and have changed my review to 'accept'.


Published as a conference paper at ICLR 2018 GENERALIZING HAMILTONIAN MONTE CARLO WITH NEURAL NETWORKS
We present a general-purpose method to train Markov chain Monte Carlo kernels, parameterized by deep neural networks, that converge and mix quickly to their target distribution. Our method generalizes Hamiltonian Monte Carlo and is trained to maximize expected squared jumped distance, a proxy for mixing speed. We demonstrate large empirical gains on a collection of simple but challenging distributions, for instance achieving a 106× improvement in effective sample size in one case, and mixing when standard HMC makes no measurable progress in a second. Finally, we show quantitative and qualitative gains on a real-world task: latent-variable generative modeling. We release an open source TensorFlow implementation of the algorithm.

The paper introduces a non-volume-preserving generalization of HMC whose transitions are determined by a set of neural network functions. These functions are trained to maximize expected squared jump distance.
This works because each variable (of the state space) is modified in turn, so that the resulting update is invertible, with a tractable transformation inspired by Dinh et al 2016.
Overall, I believe this paper is of good quality, clearly and carefully written, and potentially accelerates mixing in a state-of-the-art MCMC method, HMC, in many practical cases. A few downsides are commented on below.
The experimental section proves the usefulness of the method on a range of relevant test cases; in addition, an application to a latent variable model is provided sec5.2.
Fig 1a presents results in terms of numbers of gradient evaluations, but I couldn't find much in the way of computational cost of L2HMC in the paper. I can't see where the number "124x" in sec 5.1 stems from. As a user, I would be interested in the typical computational cost of both "MCMC sampler training" and MCMC sampler usage (inference?), compared to competing methods. This is admittedly hard to quantify objectively, but just an order of magnitude would be helpful for orientation.
Would it be relevant, in sec5.1, to compare to other methods than just HMC, eg LAHMC?
I am missing an intuition for several things: eq7, the time encoding defined in Appendix C
Appendix Fig5, I cannot quite see how the caption claim is supported by the figure (just hardly for VAE, but not for HMC).
The number "124x ESS" in sec5.1 seems at odds with the number in the abstract, "50x".
# Minor errors
- sec1: "The sampler is trained to minimize a variation": should be maximize
"as well as on a the real-world"
- sec3.2 "and 1/2 v^T v the kinetic": "energy" missing
- sec4: the acronym L2HMC is not expanded anywhere in the paper
The sentence "We will denote the complete augmented...p(d)" might be moved to after "from a uniform distribution" in the same paragraph.
In paragraph starting "We now update x":
    - specify for clarity: "the first update, which yields x' "/ "the second update, which yields x''  "
    - "only affects $x_{\bar{m}^t}$": should be $x'_{\bar{m}^t}$  (prime missing)
    - the syntax using subscript m^t is confusing to read; wouldn't it be clearer to write this as a function, eg "mask(x',m^t)"?
    - inside zeta_2 and zeta_3, do you not mean $m^t" and $\bar{m}^t$ ?
- sec5: add reference for first mention of "A NICE MC"
- Appendix A:
    - "Let's" -> "Let"
    - eq12 should be x''=...
- Appendix C: space missing after "Section 5.1"
- Appendix D1: "In this section is presented" : sounds odd
- Appendix D3: presumably this should consist of the figure 5 ? Maybe specify.


Workshop track -ICLR 2018 BENEFITS OF DEPTH FOR LONG-TERM MEMORY OF RECURRENT NETWORKS
The key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ever-improving ability to model intricate long-term temporal dependencies. However, a well established measure of RNNs' long-term memory capacity is lacking, and thus formal understanding of their ability to correlate data throughout time is limited. Though depth efficiency in convolutional networks is well established by now, it does not suffice in order to account for the success of deep RNNs on inputs of varying lengths, and the need to address their 'time-series expressive power' arises.
In this paper, we analyze the effect of depth on the ability of recurrent networks to express correlations ranging over long time-scales. To meet the above need, we introduce a measure of the information flow across time that can be supported by the network, referred to as the Start-End separation rank. Essentially, this measure reflects the distance of the function realized by the recurrent network from a function that models no interaction whatsoever between the beginning and end of the input sequence. We prove that deep recurrent networks support StartEnd separation ranks which are exponentially higher than those supported by their shallow counterparts. Moreover, we show that the ability of deep recurrent networks to correlate different parts of the input sequence increases exponentially as the input sequence extends, while that of vanilla shallow recurrent networks does not adapt to the sequence length at all. Thus, we establish that depth brings forth an overwhelming advantage in the ability of recurrent networks to model long-term dependencies, and provide an exemplar of quantifying this key attribute which may be readily extended to other RNN architectures of interest, e.g. variants of LSTM networks. We obtain our results by considering a class of recurrent networks referred to as Recurrent Arithmetic Circuits (RACs), which merge the hidden state with the input via the Multiplicative Integration operation.

After reading the authors's rebuttal I increased my score from a 7 to a 6.  I do think the paper would benefit from experimental results, but agree with the authors that the theoretical results are non-trivial and interesting on their own merit.
------------------------
The paper presents a theoretical analysis of depth in RNNs (technically a variant called RACs) i.e. stacking RNNs on top of one another, so that h_t^l (i.e. hidden state at time t and layer l is a function of h_t^{l-1} and h_{t-1}^{l})
The work is inspired by previous results for feed forward nets and CNNs. However, what is unique to RNNs is their ability to model long term dependencies across time.
To analyze this specific property, the authors propose a concept called "start-end rank" that essentially models the richness of the dependency between two disjoint subsets of inputs. Specifically, let S = {1, . . . , T/2} and E === {T/2 + 1, . . . , T}. sep_{S,E}(y) models the dependence between these two sets of time points. Specifically sep_{S,E}(y) = K means there exists g_s^k and g_e^k for k=1...K such that y(x) = \sum_{k} g_s^k(x_S) g_e^k(x_E).
Therefore sep_{S,E}(y) is the rank of a particular matricization of y (with respect to the partition S,E). If sep_{S,E}=1 then it is rank 1 (and would correspond to independence if y(x) was a probability distribution). A higher rank would correspond to more dependence across time.
(Comment: I believe if I understood the above correctly, it would be easier to explain tensors/matricization first and then introduce separation rank, since I think it much makes it clearer to explain. Right now the authors explain separation rank first and then discuss tensors / matricization).
Using this concept, the authors prove that deep recurrent networks can express functions that have exponentially higher start/end ranks than shallow RNNs.
I overall like the paper's theoretical results, but I have the following complaints:
(1)  I have the same question as the other reviewer. Why is Theorem 1 not a function of L?  Do the papers that prove similar theorems about ConvNets able to handle general L? What makes this more challenging? I feel if comparing L=2 vs L=3 is hard, the authors should be more up front about that in the introduction/abstract.
(2) I think it would have been stronger if the authors would have provided some empirical results validating their claims.


We study the problem of building models that disentangle independent factors of variation. Such models encode features that can efficiently be used for classification and to transfer attributes between different images in image synthesis. As data we use a weakly labeled training set, where labels indicate what single factor has changed between two data samples, although the relative value of the change is unknown. This labeling is of particular interest as it may be readily available without annotation costs. We introduce an autoencoder model and train it through constraints on image pairs and triplets. We show the role of feature dimensionality and adversarial training theoretically and experimentally. We formally prove the existence of the reference ambiguity, which is inherently present in the disentangling task when weakly labeled data is used. The numerical value of a factor has different meaning in different reference frames. When the reference depends on other factors, transferring that factor becomes ambiguous. We demonstrate experimentally that the proposed model can successfully transfer attributes on several datasets, but show also cases when the reference ambiguity occurs.

The paper considers the challenges of disentangling factors of variation in images: for example disentangling viewpoint from vehicle type in an image of a car. They identify a well-known problem, which they call "reference ambiguity", and show that in general without further assumptions one cannot tell apart two different factors of variation.
They then go on to suggest an interesting AE+GAN architecture where the main novelty is the idea of taking triplets such that the first two instances vary in only one factor of variation, while the third instance varies in both from the pair. This is clever and allows them to try and disentangle the variation factors using a joint encoder-decoder architecture working on the triplet.
Pros:
1. Interesting use of constructed triplets.
2. Interesting use of GAN on the artificial instance named x_{3 \oplus 1}
Cons:
1. Lack of clarity: the paper is hard to follow at times. It's not entirely obvious how the theoretical part informs the practical part. See detailed comments below.
2. The theory addresses two widely recognized problems as if they're novel:  "reference ambiguity" and "shortcut problem". The second merely refers to the fact that unconstrained autoencoders will merely memorize the instance.
3. Some of the architectural choices (the one derived from "shortcut problem") are barely explained or looked into.
Specific comments:
1. An important point regarding the reference ambiguity problem and eq. (2): a general bijective function mixing v and c would not have the two components as independent. The authors could have used this extremely important aspect of the generative process they posit in order to circumvent the problem of ambiguity. In fact, I suspect that this is what allows their method to succeed.
2. I think the intro could be made better if more concrete examples be made earlier on. Specifically the car-type/viewpoint example, along with noting what weak labels mean in that context.
3. In presenting autoencoders it is crucial to note that they are all built around the idea of compression. Otherwise, the perfect latent representation is z=x.
4. I would consider switching the order of sections 2 and 3, so the reader will be better grounded in what this paper is about before reading the related work.
5. In discussing attributes and "valid" features, I found the paper rather vague. An image has many attributes: the glint in the corner of a window, the hue of a leaf. The authors should be much more specific in this discussion and definite explicitly and clearly what they mean when they use these terms.
6. In equation (5), should it be p(v_1,v_2)? Or are v_1 and v_2 assumed to be independent?
7. Under equation (5), the paper mentions an "autoencoder constraint". Such a constraint is not mentioned up to this point in the paper if I'm not mistaken.
8. Also under equation (5): is this where the encoder requirements are defined? If so, please be more explicit about it. Also note that you should require c_1 \neq c_2.
9. In proof of Proposition 1, there is discussion of N_c. N_c was mentioned before but never properly defined; same for R_c and C^-1. These should be part of the proposition statement or defined formally. Currently they are only discussed ad-hoc after equation (5).
10 .In the proof of Proposition 1, what is f_c^-1 ? It's only defined later in the paper.
11. In general, what promises that f_c^-1 and f_v^-1 are well defined? Are f_c and f_v injective? Why?
12. Before explaining the training of the model, the task should be defined properly. What is the goal of the training?
13. In eq. (15) I am missing a term which addresses "the shortcut problem" as defined in the previous page.
14. The weak labels are never properly defined and are discussed in a vague manner. Please define what does that term mean in your context and what were the weak labels in each experiment.
15. In the conclusion, I would edit to say the "our trained model works well on *several* datasets".
Minor comments:
Please use \citep when appropriate. Instead of "Generative Adversarial Nets Goodfellow et al. (2014)", you should have "Generative Adversarial Nets (Goodfellow et al., 2014)"


We introduce the "inverse square root linear unit" (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noiserobust deactivation state, lessening the over fitting risk. The significant performance advantage of ISRLU on traditional CPUs also carry over to more efficient HW implementations on HW/SW codesign for CNNs/RNNs. In experiments with TensorFlow, ISRLU leads to faster learning and better generalization than ReLU on CNNs. This work also suggests a computationally efficient variant called the "inverse square root unit" (ISRU) which can be used for RNNs. Many RNNs use either long short-term memory (LSTM) and gated recurrent units (GRU) which are implemented with tanh and sigmoid activation functions. ISRU has less computational complexity but still has a similar curve to tanh and sigmoid.

Summary:
- The paper proposes a new activation function that looks similar to ELU but much cheaper by using the inverse square root function.
Contributions:
- The paper proposes a cheaper activation and validates it with an MNIST experiment. The paper also shows major speedup compared to ELU and TANH (unit-wise speedup).
Pros:
- The proposed function has similar behavior as ELU but 4x cheaper.
- The authors also refer us to faster ways to compute square root functions numerically, which can be of general interests to the community for efficient network designs in the future.
- The paper is clearly written and key contributions are well present.
Cons:
- Clearly, the proposed function is not faster than ReLU. In the introduction, the authors explain the motivation that ReLU needs centered activation (such as BN). But the authors also need to justify that ISRLU (or ELU) doesn’t need BN. In fact, in a recent study of ELU-ResNet (Shah et al., 2016) finds that ELU without BN leads to gradient explosion. To my knowledge, BN (at least in training time) is much more expensive than the activation function itself, so the speedup get from ISRLU may be killed by using BN in deeper networks on larger benchmarks. At inference time, all of ReLU, ELU, and ISRLU can fuse BN weights into convolution weights, so again ISRLU will not be faster than ReLU. The core question here is, whether the smoothness and centered zero property of ELU can buy us any win, compared to ReLU? I couldn’t find it based on the results presented here.
- The authors need to validate on larger datasets (e.g. CIFAR, if not ImageNet) so that their proposed methods can be widely adopted.
- The speedup is only measured on CPU. For practical usage, especially in computer vision, GPU speedup is needed to show an impact.
Conclusion:
- Based on the comments above, I recommend weak reject.
References:
- Shah, A., Shinde, S., Kadam, E., Shah, H., Shingade, S.. Deep Residual Networks with Exponential Linear Unit. In Proceedings of the Third International Symposium on Computer Vision and the Internet (VisionNet'16).


We propose a novel hierarchical generative model with a simple Markovian structure and a corresponding inference model. Both the generative and inference model are trained using the adversarial learning paradigm. We demonstrate that the hierarchical structure supports the learning of progressively more abstract representations as well as providing semantically meaningful reconstructions with different levels of fidelity. Furthermore, we show that minimizing the Jensen-Shanon divergence between the generative and inference network is enough to minimize the reconstruction error. The resulting semantically meaningful hierarchical latent structure discovery is exemplified on the CelebA dataset. There, we show that the features learned by our model in an unsupervised way outperform the best handcrafted features. Furthermore, the extracted features remain competitive when compared to several recent deep supervised approaches on an attribute prediction task on CelebA. Finally, we leverage the model's inference network to achieve state-ofthe-art performance on a semi-supervised variant of the MNIST digit classification task.

_________________________________________________________________________________________________________
I raise my rating on the condition that the authors will also address the minor concerns in the final version, please see details below.
_________________________________________________________________________________________________________
This paper proposes to perform Adversarially Learned Inference (ALI) in a layer-wise manner. The idea is interesting, and the authors did a good job to describe high-level idea, and demonstrate one advantage of hierarchy: providing different levels reconstructions. However, the advantage of better reconstruction could be better demonstrated.  Some major concerns should be clarified before publishing:
(1) How did the authors implement p(x|z) and q(z|x), or p(z_l | z_{l+1}) and q(z_{l+1} | z_l )? Please provide the details, as this is key to the reconstruction issues of ALI.
(2) Could the authors provide the pseudocode procedure of the proposed algorithm? In the current form of the writing, it is not clear what the HALI procedure is, whether (1) one discriminator is used to distinguish the concatenation of (x, z_1, ..., z_L), or (2) L discriminators are used to distinguish the concatenation of (z_l, z_{l+1}) at each layer, respectively?
The above two points are important. If not correctly constructed, it might reveal potential flaws of the proposed technique.
Since one of the major claims for HALI is to provide better reconstruction with higher fidelity than ALI. Could the authors provide quantitative results on MNIST and CIFAR to demonstrate this? The reconstruction issues have first been highlighted and theoretically analyzed in ALICE [*], and some remedy has been proposed to alleviate the issue.  Quantitative comparison on MNIST and CIFAR are also conducted. Could the authors report numbers to compare with them (ALI and ALICE)?
The 3rd paragraph in Introduction should be adjusted to correctly clarify details of algorithms, and reflect up-to-date literature. "One interesting feature highlighted in the original ALI work (Dumoulin et al., 2016) is that ... never explicitly trained to perform reconstruction, this can nevertheless be easily done...". Note that ALI can only perform reconstruction when the deterministic mapping is used, while ALI itself adopted the stochastic mapping. Further, the deterministic mapping is the major difference of BiGAN from ALI. Therefore, more rigorous way to phrase is that "the original ALI work with deterministic mappings", or "BiGAN" never explicitly trained to perform reconstruction, this can nevertheless be easily done... This tiny difference between deterministic/stochastic mappings makes major difference for the quality of reconstruction, as theoretically analyzed and experimentally compared in ALICE. In ALICE, the authors confirmed further source of poor reconstructions of ALI in practice. It would be better to reflect the non-identifiability issues raised by ALICE in Introduction, rather than hiding it in Future Work as "Although recent work designed to improve the stability of training in ALI does show some promise (Chunyuan Li, 2017), more work is needed on this front."
Also, please fix the typo in reference as:
[*] Chunyuan Li, Hao Liu, Changyou Chen, Yunchen Pu, Liqun Chen, Ricardo Henao and Lawrence Carin. ALICE: Towards understanding adversarial learning for joint distribution matching. In Advances in Neural Information Processing Systems (NIPS), 2017.


Robust real-world learning should benefit from both demonstrations and interaction with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on reward from the environment. These tasks have divergent losses which are difficult to jointly optimize; further, such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstration and refines the policy in a real environment. Crucially, both learning from demonstration and interactive refinement use exactly the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data, since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

SUMMARY:
The motivation for this work is to have an RL algorithm that can use imperfect demonstrations to accelerate learning. The paper proposes an actor-critic algorithm, called Normalized Actor-Critic (NAC), based on the entropy-regularized formulation of RL, which is defined by adding the entropy of the policy as an additional term in the reward function.
Entropy-regularized formulation leads to nice relationships between the value function and the policy, and has been explored recently by many, including [Ziebart, 2010], [Schulman, 2017], [Nachum, 2017], and [Haarnoja, 2017].
The paper benefits from such a relationship and derives an actor-critic algorithm. Specifically, the paper only parametrizes the Q function, and computes the policy gradient using the relation between the policy and Q function (Appendix A.1).
Through a set of experiments, the paper shows the effectiveness of the method.
EVALUATION:
I think exploring and understanding entropy-regularized RL algorithm is important. It is also important to be able to benefit from off-policy data. I also find the empirical results encouraging. But I have some concerns about this paper:
- The derivations of the paper are unclear.
- The relation with other recent work in entropy-regularized RL should be expanded.
- The work is less about benefiting from demonstration data and more about using off-policy data.
- The algorithm that performs well is not the one that was actually derived.
* Unclear derivations:
The derivations of Appendix A.1 is unclear. It makes it difficult to verify the derivations.
To begin with, what is the loss function of which (9) and (10) are its gradients?
To be more specific, the choices of \hat{Q} in (15) and \hat{V} in (19) are not clear.  For example, just after (18) it is said that “\hat{Q} could be obtained through bootstrapping by R + gamma V_Q”. But if it is the case, shouldn’t we have a gradient of Q in (15) too? (or show that it can be ignored?)
It appears that \hat{Q} and \hat{V} are parameterized independently from Q (which is a function of theta). Later in the paper they are estimated using a target network, but this is not specified in the derivations.
The main problem boils down to the fact that the paper does not start from a loss function and compute all the gradients in a systematic way. Instead it starts from gradient terms, each of which seems to be from different papers, and then simplifies them. For example, the policy gradient in (8), which is further decomposed in Appendix A.1 as (15) and (16) and simplified, appears to be Eq. (50) of [Schulman et al., 2017] (https://arxiv.org/abs/1704.06440). In that paper we have Q_pi instead of \hat{Q} though.
I suggest that the authors start from a loss function and clearly derive all necessary steps.
* Unclear relation with other papers:
What part of the derivations of this work are novel? Currently the novelty is not obvious.
For example, having the gradient of both Q and V, as in (9), has been stated by [Haarnoja et al., 2017] (very similar formulation is developed in Appendix B of https://arxiv.org/abs/1702.08165).
An algorithm that can work with off-policy data has also been developed by [Nachum, 2017] (in the form of a Bellman residual minimization algorithm, as opposed to this work which essentially uses a Fitted Q-Iteration algorithm as the critic).
I think the paper could do a better job differentiating from those other papers.
* The claim that this paper is about learning from demonstration is a bit questionable. The paper essentially introduces a method to use off-policy data, which is of course important, but does not cover the important scenario where we only have access to (state,action) pairs given by an expert. Here it appears from the description of Algorithm 1 that the transitions in the demonstration data have the same semantic as the interaction data, i.e., (s,a,r,s’).
This makes it different from the work by [Kim et al., 2013], [Piot et al., 2014], and [Chemali et al., 2015], which do not require such a restriction on the demonstration data.
* The paper mentions that to formalize the method as a policy gradient one, importance sampling should be used (the paragraph after (12)), but the performance of such a formulation is bad, as depicted in Figure 2. As a result, Algorithm 1 does not use importance sampling.
This basically suggests that by ignoring the fact that the data is collected off-policy, and treating it as an on-policy data, the agent might perform better. This is an interesting phenomenon and deservers further study, as currently doing the “wrong” things is better than doing the “right” thing. I think a good paper should investigate this fact more.


Profiling cellular phenotypes from microscopic imaging can provide meaningful biological information resulting from various factors affecting the cells. One motivating application is drug development: morphological cell features can be captured from images, from which similarities between different drugs applied at different dosages can be quantified. The general approach is to find a function mapping the images to an embedding space of manageable dimensionality whose geometry captures relevant features of the input images. An important known issue for such methods is separating relevant biological signal from nuisance variation. For example, the embedding vectors tend to be more correlated for cells that were cultured and imaged during the same week than for cells from a different week, despite having identical drug compounds applied in both cases. In this case, the particular batch a set of experiments were conducted in constitutes the domain of the data; an ideal set of image embeddings should contain only the relevant biological information (e.g. drug effects). We develop a general framework for adjusting the image embeddings in order to 'forget' domain-specific information while preserving relevant biological information. To do this, we minimize a loss function based on distances between marginal distributions (such as the Wasserstein distance) of embeddings across domains for each replicated treatment. For the dataset presented, the replicated treatment is the negative control. We find that for our transformed embeddings (1) the underlying geometric structure is not only preserved but the embeddings also carry improved biological signal (2) less domain-specific information is present.

The authors present a method that aims to remove domain-specific information while preserving the relevant biological information between biological data measured in different experiments or "batches". A network is trained to learn the transformations that minimize the Wasserstein distance between distributions. The wasserstein distance is also called the "earth mover distance" and is traditionally formulated as the cost it takes for an optimal transport plan to move one distribution to another. In this paper they have a neural network compute the wasserstein distance using a different formulation that was used in Arjovsky et al. 2017, finds a lipschitz function f, which shows the maximal difference when evaluated on samples from the two distributions. Here these functions are formulated as affine transforms of the data with parameters theta that are computed by a neural network. Results are examined mainly by looking at the first two PCA components of the data.
The paper presents an interesting idea and is fairly well written. However I have a few concerns:
1. Most of the ideas presented in the paper rely on works by Arjovsky et al. (2017), Gulrajani et al. (2017), and Gulrajani et al. (2017). Some selections, which are presented in the papers are not explained, for example, the gradient penalty, the choice of \lambda and the choice of points for gradient computation.
2. The experimental results are not fully convincing, they simply compare the first two PC components on this Broad Bioimage benchmark collection. This section could be improved by demonstrating the approach on more datasets.
3. There is a lack comparison to other methods such as Shaham et al. (2017). Why is using earth mover distance better than MMD based distance? They only compare it to a method named CORAL and to Typical
Variation Normalization (TVN). What about comparison to other batch normalization methods in biology such as SEURAT?
4. Why is the affine transform assumption valid in biology? There can definitely be non-linear effects that are different between batches, such as ion detection efficiency differences.
5. Only early stopping seems to constrain their model to be near identity. Doesn't this also prevent optimal results ? How does this compare to the near-identity constraints in resnets in Shaham et al. ?


KRONECKER-FACTORED CURVATURE APPROXIMA- TIONS FOR RECURRENT NEURAL NETWORKS
Kronecker-factor Approximate Curvature (Martens & Grosse, 2015) (K-FAC) is a 2nd-order optimization method which has been shown to give state-of-the-art performance on large-scale neural network optimization tasks (Ba et al., 2017). It is based on an approximation to the Fisher information matrix (FIM) that makes assumptions about the particular structure of the network and the way it is parameterized. The original K-FAC method was applicable only to fully-connected networks, although it has been recently extended by Grosse & Martens (2016) to handle convolutional networks as well. In this work we extend the method to handle RNNs by introducing a novel approximation to the FIM for RNNs. This approximation works by modelling the statistical structure between the gradient contributions at different time-steps using a chain-structured linear Gaussian graphical model, summing the various cross-moments, and computing the inverse in closed form. We demonstrate in experiments that our method significantly outperforms general purpose state-of-the-art optimizers like SGD with momentum and Adam on several challenging RNN training tasks.

This paper extends the Kronecker-factor Approximate Curvature (K-FAC) optimization method to the setting of recurrent neural networks. The K-FAC method is an approximate 2nd-order optimization method that builds a block diagonal approximation of the Fisher information matrix, where the block diagonal elements are Kronecker products of smaller matrices.
In order to approximate the Fisher information matrix for RNNs, the authors assume that the derivative of the loss function with respect to each weight matrix at each time step is independent of the length of the sequence, that these derivatives are temporally homogeneous, that the input and derivatives of the output are independent across every point in time, and that either the one-step cross-covariance of these derivatives is symmetric or that the training sequences are effectively infinite in length. Based on these assumptions, the authors show that the Fisher information can be reduced into a form in which the derivatives of the weight matrices can be approximated by a linear Gaussian graphical model and in which the approximate 2nd order method can be efficiently carried out. The authors compare their method to SGD on two language modeling tasks and against Adam for learning differentiable neural computers.
The paper is relatively clear, and the authors do a reasonable job of introducing related work of the original K-FAC algorithm as well as its extension to CNNs before systematically deriving their method for RNNs. The problem of extending the K-FAC algorithm is natural, and the steps taken in this paper seem natural yet also original and non-trivial.
The main issue that I have with this paper is the lack of theoretical justification or even intuition for the many approximations carried out in the course of approximating the Fisher information matrix. In many instances, it seemed like these approximations were made purely for convenience and tractability without much regard for (even approximate) correctness. This quality of this paper would be greatly  strengthened if it had some bounds on approximation error or even empirical results testing the validity of the assumptions in the paper. Moreover, the experiments do not demonstrate levels of statistical significance in the results, so it is difficult to assert the practical significance of this work.
Specific comments and questions
Page 2, "r is is". Typo.
Page 2, "DV". I found the introduction of V without any explanation to be confusing.
Page 2, "P_{y|x}(\theta)". The relation between P_{y|x}(\theta) and f(x,\theta) is never explained.
Page 3, "common practice of computing the natural gradient as (F + \lambda I) \nabla h instead of F^{-1} \nabla h". I don't see how the former can serve as a replacement for the latter.
Page 3, "approximate g and a as statistically independent". Even though K-FAC already exists, it would be good to explain why this assumption is reasonable, since similar assumptions are made for the work presented in this paper.
Page 4, "This new approximation, called "KFC", is derived by assuming....". Same as previous comment. It would be good to briefly discuss why these assumptions are reasonable.
Page 5, Independence of T and w_t's, temporal homogeneity of w_t's,, and independence between a_t's and g_t's. I can see why these are convenient assumptions, but why are they reasonable? Moreover, why is it further natural to assume that A and G are temporally homogeneous as well?
Page 7, "But insofar as the w_t's ... encode the relevant information contained in these external variables, they should be approximately Markovian". I am not sure what this means.
Page 7, "The linear-Gaussian assumption meanwhile is a more severe one to make, but it seems necessary for there to be any hope that the required expectations remain tractable". I am not sure that this is a good enough justification for such an idea, unless there are compelling approximation error bounds.
Page 8, Option 1. In what situations is it reasonable to assume that V_1 is symmetric?
Pages 8-9, Option 2. What is a good finite sample size in which the assumption that the training sequences are infinitely long is reasonable in practice? Can the error |\kappa(x) - \zeta_T(x)| be translated into a statement on the approximation error?
Page 9, "V_1 = V_{1,0} = ...". Typos (that appear to have been caught by the authors already).
Page 9, "The 2nd-order statistics ... are accumulated through an exponential moving average during training". How sensitive is the performance of this method to the decay rate of the exponential moving average?
Page 10, "The additional computations required to get the approximate Fisher inverse from these statistics ... are performed asynchronously on the CPU's". I find it a bit unfair to compare SGD to K-FAC in terms of wall clock time without also using the extra CPU's for SGD as well (e.g. via Hogwild or synchronous parallel SGD).
Page 10, "The hyperparameters of our approach...". What is the sensitivity of the experimental results to these hyperparameters? Moreover, how sensitive are the results to initialization?
Page 10, "we found that each parameter update of our method required about 80% more wall-clock time than an SGD update". How much of this is attributed to the fact that the statistics are computed asynchronously?
Pages 10-12, Experiments. There are no error bars in any of the plots, so it is impossible to ascertain the statistical significance of any of these results.
Page 11: Figure 2. Where is the Adam batchsize 50 line in the left plot? Why did the Adam batchsize 200 line disappear halfway through the right plot?


We present a novel optimization strategy for training neural networks which we call "BitNet". The parameters of neural networks are usually unconstrained and have a dynamic range dispersed over all real values. Our key idea is to limit the expressive power of the network by dynamically controlling the range and set of values that the parameters can take. We formulate this idea using a novel end-to-end approach that regularizes a typical classification loss function. Our regularizer is inspired by the Minimum Description Length (MDL) principle. For each layer of the network, our approach optimizes real-valued translation and scaling factors and integervalued parameters (weights). We empirically compare BitNet to an equivalent unregularized model on the MNIST and CIFAR-10 datasets. We show that BitNet converges faster to a superior quality solution. Additionally, the resulting model has significant savings in memory due to the use of integer-valued parameters.

The paper proposes a technique for training quantized neural networks, where the precision (number of bits) varies per layer and is learned in an end-to-end fashion. The idea is to add two terms to the loss, one representing quantization error, and the other representing the number of discrete values the quantization can support (or alternatively the number of bits used). Updates are made to the parameter representing the # of bits via the sign of its gradient. Experiments are conducted using a LeNet-inspired architecture on MNIST and CIFAR10.
Overall, the idea is interesting, as providing an end-to-end trainable technique for distributing the precision across layers of a network would indeed be quite useful. I have a few concerns: First, I find the discussion around the training methodology insufficient. Inherently, the objective is discontinuous since # of bits is a discrete parameter. This is worked around by updating the parameter using the sign of its gradient. This is assuming the local linear approximation given by the derivative is accurate enough one integer away; this may or may not be true, but it's not clear and there is little discussion of whether this is reasonable to assume.
It's also difficult for me to understand how this interacts with the other terms in the objective (quantization error and loss). We'd like the number of bits parameter to trade off between accuracy (at least in terms of quantization error, and ideally overall loss as well) and precision. But it's not at all clear that the gradient of either the loss or the quantization error w.r.t. the number of bits will in general suggest increasing the number of bit (thus requiring the bit regularization term). This will clearly not be the case when the continuous weights coincide with the quantized values for the current bit setting. More generally, the direction of the gradient will be highly dependent on the specific setting of the current weights. It's unclear to me how effectively accuracy and precision are balanced by this training strategy, and there isn't any discussion of this point either.
I would be less concerned about the above points if I found the experiments compelling. Unfortunately, although I am quite sympathetic to the argument that state of the art results or architectures aren't necessary for a paper of this kind, the results on MNIST and CIFAR10 are so poor that they give me some concern about how the training was performed and whether the results are meaningful. Performance on MNIST in the 7-11% test error range is comparable to a simple linear logistic regression model; for a CNN that is extremely bad. Similarly, 40% error on CIFAR10 is worse than what some very simple fully connected models can achieve.
Overall, while I like the and think the goal is good, I think the motivation and discussion for the training methodology is insufficient, and the empirical work is concerning. I can't recommend acceptance.


Published as a conference paper at ICLR 2018 META-LEARNING AND UNIVERSALITY: DEEP REPRESENTATIONS AND GRADIENT DESCENT CAN APPROXIMATE ANY LEARNING ALGORITHM
Learning to learn is a powerful paradigm for enabling models to learn from data more effectively and efficiently. A popular approach to meta-learning is to train a recurrent model to read in a training dataset as input and output the parameters of a learned model, or output predictions for new test inputs. Alternatively, a more recent approach to meta-learning aims to acquire deep representations that can be effectively fine-tuned, via standard gradient descent, to new tasks. In this paper, we consider the meta-learning problem from the perspective of universality, formalizing the notion of learning algorithm approximation and comparing the expressive power of the aforementioned recurrent models to the more recent approaches that embed gradient descent into the meta-learner. In particular, we seek to answer the following question: does deep representation combined with standard gradient descent have sufficient capacity to approximate any learning algorithm? We find that this is indeed true, and further find, in our experiments, that gradient-based meta-learning consistently leads to learning strategies that generalize more widely compared to those represented by recurrent models.

This paper studies the capacity of the model-agnostic meta-learning (MAML) framework as a universal learning algorithm approximator. Since a (supervised) learning algorithm can be interpreted as a map from a dataset and an input to an output, the authors define a universal learning algorithm approximator to be a universal function approximator over the set of functions that map a set of data points and an  input to an output. The authors show constructively that there exists a neural network architecture for which the model learned through MAML can approximate any learning algorithm.
The paper is for the most part clear, and the main result seems original and technically interesting. At the same time, it is not clear to me that this result is also practically significant. This is because the universal approximation result relies on a particular architecture that is not necessarily the design one would always use in MAML. This implies that MAML as typically used (including in the original paper by Finn et al, 2017a) is not necessarily a universal learning algorithm approximator, and this paper does not actually justify its empirical efficacy theoretically. For instance, the authors do not even use the architecture proposed in their proof in their experiments. This is in contrast to the classical universal function approximator results for feedforward neural networks, as a single hidden layer feedforward network is often among the family of architectures considered in the course of hyperparameter tuning. This distinction should be explicitly discussed in the paper.  Moreover, the questions posed in the experimental results do not seem related to the theoretical result, which seems odd.
Specific comments and questions:
Page 4: "\hat{f}(\cdot; \theta') approximates f_{\text{target}}(x, y, x^*) up to arbitrary position". There seems to be an abuse of notation here as the first expression is a function and the second expression is a value.
Page 4: "to show universality, we will construct a setting of the weight matrices that enables independent control of the information flow...". How does this differ from the classical UFA proofs? The relative technical merit of this paper would be more clear if this is properly discussed.
Page 4: "\prod_{i=1}^N (W_i - \alpha \nabla_{W_i})". There seems to be a typo here: \nabla_{W_i} should be \nabla_{W_i} L.
Page 7: "These error functions effectively lose information because simply looking at their gradient is insufficient to determine the label." It would be interesting the compare the efficacy of MAML on these error functions as compared to cross entropy and mean-squared error.
Page 7: "(1) can a learner trained with MAML further improve from additional gradient steps when learning new tasks at test time...? (2) does the inductive bias of gradient descent enable better few-shot learning performance on tasks outside of the training distribution...?". These questions seem unrelated to the universal learning algorithm approximator result that constitutes the main part of the paper. If you're going to study these question empirically, why didn't you also try to investigate them theoretically (e.g. sample complexity and convergence of MAML)? A systematic and comprehensive analysis of these questions from both a theoretical and empirical perspective would have constituted a compelling paper on its own.
Pages 7-8: Experiments. What are the architectures and hyperparameters used in the experiments, and how sensitive are the meta-learning algorithms to their choice?
Page 8: "our experiments show that learning strategies acquired with MAML are more successful when faced with out-of-domain tasks compared to recurrent learners....we show that the representations acquired with MAML are highly resilient to overfitting". I'm not sure that such general claims are justified based on the experimental results in this paper. Generalizing to out-of-domain tasks is heavily dependent on the specific level and type of drift between the old and new distributions. These properties aren't studied at all in this work.
POST AUTHOR REBUTTAL: After reading the response from the authors and seeing the updated draft, I have decided to upgrade my rating of the manuscript to a 6. The universal learning algorithm approximator result is a nice result, although I do not agree with the other reviewer that it is a  "significant contribution to the theoretical understanding of meta-learning," which the authors have reinforced (although it can probably be considered a significant contribution to the theoretical understanding of MAML in particular). Expressivity of the model or algorithm is far from the main or most significant consideration in a machine learning problem, even in the standard supervised learning scenario. Questions pertaining to issues such as optimization and model selection are just as, if not more, important. These sorts of ideas are explored in the empirical part of the paper, but I did not find the actual experiments in this section to be very compelling. Still, I think the universal learning algorithm approximator result is sufficient on its own for the paper to be accepted.


DEEP GAUSSIAN EMBEDDING OF GRAPHS: UNSUPERVISED INDUCTIVE LEARNING VIA RANKING
Methods that learn representations of nodes in a graph play a critical role in network analysis since they enable many downstream learning tasks. We propose Graph2Gauss -an approach that can efficiently learn versatile node embeddings on large scale (attributed) graphs that show strong performance on tasks such as link prediction and node classification. Unlike most approaches that represent nodes as point vectors in a low-dimensional continuous space, we embed each node as a Gaussian distribution, allowing us to capture uncertainty about the representation. Furthermore, we propose an unsupervised method that handles inductive learning scenarios and is applicable to different types of graphs: plain/attributed, directed/undirected. By leveraging both the network structure and the associated node attributes, we are able to generalize to unseen nodes without additional training. To learn the embeddings we adopt a personalized ranking formulation w.r.t. the node distances that exploits the natural ordering of the nodes imposed by the network structure. Experiments on real world networks demonstrate the high performance of our approach, outperforming state-of-the-art network embedding methods on several different tasks. Additionally, we demonstrate the benefits of modeling uncertainty -by analyzing it we can estimate neighborhood diversity and detect the intrinsic latent dimensionality of a graph.

This paper proposes Graph2Gauss (G2G), a node embedding method that embeds nodes in attributed graphs (can work w/o attributes as well) into Gaussian distributions rather than conventionally latent vectors. By doing so, G2G can reflect the uncertainty of a node's embedding. The authors then use these Gaussian distributions and neighborhood ranking constraints to obtain the final node embeddings. Experiments on link prediction and node classification showed improved performance over several strong embedding methods. Overall, the paper is well-written and the contributions are remarkable. The reason I am giving a less possible rating is that some statements are questionable and can severely affect the conclusions claimed in this paper, which therefore requires the authors' detailed response. I am certainly willing to change my rating if the authors clarify my questions.
Major concern 1: Is the latent vector dimension L really the same for G2G and other compared methods?
In the first paragraph of Section 4, it is stated that "in all experiments if the competing techniques use an embedding of
dimensionality L, G2G’s embedding is actually only half of this dimensionality so that the overall number of ’parameters’ per node (mean vector + variance terms) matches L."  This setting can be wrong since the degree of freedom of a L-dim Gaussian distribution should be L+L(L-1)/2, where the first term corresponds to the mean and the second term corresponds to the covariance. If I understand it correctly, when any compared embedding method used an L-dim vector, the authors used the dimension of L/2. But this setting is wrong if one wants the overall number of ’parameters’ per node (mean vector + variance terms) matches L, as stated by the authors. Fixing L, the equivalent dimension L_G2G for G2G should be set such that L_G2G +L_G2G (L_G2G -1)/2=L, not 2*L_G2G=L.  Since this setting is universal to the follow-up analysis and may severely degrade the performance of GSG due to less embedding dimensions, I hope the authors can clarify this point.
Major concern 2: The claim on inductive learning
Inductive learning is one of the major contributions claimed in this paper. The authors claim G2G can learn an embedding of an unseen node solely based on their attributes. However, is it not clear why this can be done. In the learning stage of Sec. 3.3, the attributes do not seem to play a role in the energy function. Also, since no algorithm descriptions are available, it's not clear how using only an unseen node's attributes can yield a good embedding under G2G work (so does Sec. 4.5).
Moreover, how does it compare to directly using raw user attributes for these tasks?
Minor concern/suggestions: The "similarity" measure in section 3.1 using KL divergence should be better rephased by "dissimilarity" measure. Otherwise, one has a similarity measure $Delta$ and wants it to increase as the hop distance k decreases (closer nodes are more similar). But the ranking constraints are somewhat counter-intuitive because you want $Delta$ to be small if nodes are closer. There is nothing wrong with the ranking condition, but rather an inconsistency between the use of "similarity" measure for KL divergence.


Published as a conference paper at ICLR 2018 LEARNING DIFFERENTIALLY PRIVATE RECURRENT LANGUAGE MODELS
We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy. Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent. In particular, we add user-level privacy protection to the federated averaging algorithm, which makes "large step" updates from user-level data. Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work. We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset.

Summary of the paper
-------------------------------
The authors propose to add 4 elements to the 'FederatedAveraging' algorithm to provide a user-level differential privacy guarantee. The impact of those 4 elements on the model'a accuracy and privacy is then carefully analysed.
Clarity, Significance and Correctness
--------------------------------------------------
Clarity: Excellent
Significance: I'm not familiar with the literature of differential privacy, so I'll let more knowledgeable reviewers evaluate this point.
Correctness: The paper is technically correct.
Questions
--------------
1. Figure 1: Adding some noise to the updates could be view as some form of regularization, so I have trouble understand why the models with noise are less efficient than the baseline.
2. Clipping is supposed to help with the exploding gradients problem. Do you have an idea why a low threshold hurts the performances? Is it because it reduces the amplitude of the updates (and thus simply slows down the training)?
3. Is your method compatible with other optimizers, such as RMSprop or ADAM (which are commonly used to train RNNs)?
Pros
------
1. Nice extensions to FederatedAveraging to provide privacy guarantee.
2. Strong experimental setup that analyses in details the proposed extensions.
3. Experiments performed on public datasets.
Cons
-------
None
Typos
--------
1. Section 2, paragraph 3 : "is given in Figure 1" -> "is given in Algorithm 1"
Note
-------
Since I'm not familiar with the differential privacy literature, I'm flexible with my evaluation based on what other reviewers with more expertise have to say.


Published as a conference paper at ICLR 2018 LEARNING ROBUST REWARDS WITH ADVERSARIAL INVERSE REINFORCEMENT LEARNING
Reinforcement learning provides a powerful and general framework for decision making and control, but its application in practice is often hindered by the need for extensive feature and reward engineering. Deep reinforcement learning methods can remove the need for explicit engineering of policy or value features, but still require a manually specified reward function. Inverse reinforcement learning holds the promise of automatic reward acquisition, but has proven exceptionally difficult to apply to large, high-dimensional problems with unknown dynamics. In this work, we propose AIRL, a practical and scalable inverse reinforcement learning algorithm based on an adversarial reward learning formulation. We demonstrate that AIRL is able to recover reward functions that are robust to changes in dynamics, enabling us to learn policies even under significant variation in the environment seen during training. Our experiments show that AIRL greatly outperforms prior methods in these transfer settings.

SUMMARY:
This paper considers the Inverse Reinforcement Learning (IRL) problem, and particularly suggests a method that obtains a reward function that is robust to the change of dynamics of the MDP.
It starts from formulating the problem within the MaxEnt IRL framework of Ziebart et al. (2008). The challenge of MaxEnt IRL is the computation of a partition function. Guided Cost Learning (GCL) of Finn et al. (2016b) is an approximation of MaxEnt IRL that uses an adaptive importance sampler to estimate the partition function. This can be shown to be a form of GAN, obtained by using a specific discriminator [Finn et al. (2016a)].
If the discriminator directly works with trajectories tau, the result would be GAN-GCL. But this leads to high variance estimates, so the paper suggests using a single state-action formulation, in which the discriminator f_theta(s,a) is a function of (s,a) instead of the trajectory. The optimal solution of this discriminator is to have f(s,a) = A(s,a) — the advantage function.
The paper, however, argues that the advantage function is “entangled” with the dynamics, and this is undesirable. So it modified the discriminator to learn a function that is a combination of two terms, one only depends on state-action and the other depends on state, and has the form of shaped reward transformation.
EVALUATION:
This is an interesting paper with good empirical results. As I am not very familiar with the work of Finn et al. (2016a) and Finn et al. (2016b), I have not verified the detail of derivations of this new paper very closely. That being said, I have some comments and questions:
* The MaxEnt IRL formulation of this work, which assumes that p_theta(tau) is proportional to exp( r_theta (tau) ), comes from
[Ziebart et al., 2008] and assumes a deterministic dynamics. Ziebart’s PhD dissertation [Ziebart, 2010] or the following paper show that the formulation is different for stochastic dynamics:
Ziebart, Bagnell, Dey, “The Principle of Maximum Causal Entropy for Estimating Interacting Processes,” IEEE Trans. on IT, 2013.
Is it still a reasonable thing to develop based on this earlier, an inaccurate, formulation?
* I am not convinced about the argument of Appendix C that shows that AIRL recovers reward up to constants.
It is suggested that since the only items on both sides of the equation on top of p. 13 depend on s’ are h* and V, they should be equal.
This would be true if s’ could be chosen arbitrararily. But s’ would be uniquely determined by s for a deterministic dynamics. In that case, this conclusion is not obvious anymore.
Consider the state space to be integers 0, 1, 2, 3, … .
Suppose the dynamics is that whenever we are at state s (which is an integer), at the next time step the state decreases toward 1, that is s’ = phi(s,a) = s - 1; unless s = 0, which we just stay at s’ = s = 0. This is independent of actions.
Also define r(s) = 1/s for s>=1 and r(0) = 0.
Suppose the discount factor is gamma = 1 (note that in Appendix B.1, the undiscounted case is studied, so I assume gamma = 1 is acceptable).
With this choices, the value function V(s) = 1/s + 1/(s-1) + … + 1/1 = H_s, i.e., the Harmonic function.
The advantage function is zero. So we can choose g*(s) = 0, and h*(s) = h*(s’) = 1.
This is in contrast to the conclusion that h*(s’) = V(s’) + c, which would be H_s + c, and g*(s) = r(s) = 1/s.
(In fact, nothing is special about this choice of reward and dynamics.)
Am I missing something obvious here?
Also please discuss how ergodicity leads to the conclusion that spaces of s’ and s are identical. What does “space of s” mean? Do you mean the support of s? Please make the argument more rigorous.
* Please make the argument of Section 5.1 more rigorous.


Many goal-oriented dialog tasks, especially ones in which the dialog system has to interact with external knowledge sources such as databases, have to handle a large number of Named Entities (NEs). There are at least two challenges in handling NEs using neural methods in such settings: individual NEs may occur only rarely making it hard to learn good representations of them, and many of the Out Of Vocabulary words that occur during test time may be NEs. Thus, the need to interact well with these NEs has emerged as a serious challenge to building neural methods for goal-oriented dialog tasks. In this paper, we propose a new neural method for this problem, and present empirical evaluations on a structured Question Answering task and three related goal-oriented dialog tasks that show that our proposed method can be effective in interacting with NEs in these settings.

The paper addresses the task of dealing with named entities in goal oriented dialog systems. Named entities, and rare words in general, are indeed troublesome since adding them to the dictionary is expensive, replacing them with coarse labels (ne_loc, unk) looses information, and so on. The proposed solution is to extend neural dialog models by introducing a named entity table, instantiated on the fly, where the keys are distributed representations of the dialog context and the values are the named entities themselves. The approach is applied to settings involving interacting to a database and a mechanism for handling the interaction is proposed. The resulting model is illustrated on a few goal-oriented dialog tasks.
I found the paper difficult to read. The concrete mappings used to create the NE keys and attention keys are missing. Providing more structure to the text would also be useful vs. long, wordy paragraphs. Here are some specific questions:
1. How are the keys generated? That are the functions used? Does the "knowledge of the current user utterance" include the word itself? The authors should include the exact model specification, including for the HRED model.
2. According to the description, referring to an existing named entity must be done by "generating a key to match the keys in the NE table and then retrieve the corresponding value and use it". Is there a guarantee that a same named entity, appearing later in the dialog, will be given the same key?  Or are the keys for already found entities retrieved directly, by value?
3. In the decoding phase, how does the system decide whether to query the DB?
4. How is the model trained?
In its current form, it's not clear how the proposed approach tackles the shortcomings mentioned in the introduction. Furthermore, while the highlighted contribution is the named entity table, it is always used in conjunction to the database approach. This raises the question whether the named entity table can only work in this context.
For the structured QA task, there are 400 training examples, and 100 named entities. This means that the number of training examples per named entity is very small. Is that correct? If yes, then it's not very surprising that adding the named entities to the vocabulary leads to overfitting. Have you compared with using random embeddings for the named entities?
Typos: page 2, second-to-last paragraph: firs -> first, page 7, second to last paragraph: and and -> and


Robustness and security of machine learning (ML) systems are intertwined, wherein a non-robust ML system (classifiers, regressors, etc.) can be subject to attacks using a wide variety of exploits. With the advent of scalable deep learning methodologies, a lot of emphasis has been put on the robustness of supervised, unsupervised and reinforcement learning algorithms. Here, we study the robustness of the latent space of a deep variational autoencoder (dVAE), an unsupervised generative framework, to show that it is indeed possible to perturb the latent space, flip the class predictions and keep the classification probability approximately equal before and after an attack. This means that an agent that looks at the outputs of a decoder would remain oblivious to an attack.

This paper is concerned with both security and machine learning.
Assuming that data is encoded, transmited, and decoded using a VAE,
the paper proposes a man-in-middle attack that alters the VAE encoding of the input data so that the decoded output will be misclassified.
The objectives are to: 1) fool the autoencoder; the classification output of the autoencoder is different from the actual class of the input; 2) make minimal change in the middle so that the attack is not detectable.
This paper is concerned with both security and machine learning, but there is no clear contributions to either field. From the machine learning perspective, the proposed "attacking" method is standard without any technical novelty. From the security perspective, the scenarios are too simplistic. The encoding-decoding mechanism being attacked is too simple without any security enhancement. This is an unrealistic scenario. For applications with security concerns, there should have been methods to guard against man-in-the-middle attack, and the paper should have at least considered some of them. Without considering the state-of-the-art security defending mechanism, it is difficult to judge the contribution of the paper to the security community.
I am not a security expert, but I doubt that the proposed method are formulated based on well founded security concepts and ideas. For example, what are the necessary and sufficient conditions for an attacking method to be undetectable? Are the criteria about the magnitude of epsilon given on Section 3.3. necessary and sufficient? Is there any reference for them? Why do we require the correspondence between the classification confidence of tranformed and original data? Would it be enough to match the DISTRIBUTION of the confidence?


SPECTRALNET: SPECTRAL CLUSTERING USING DEEP NEURAL NETWORKS
Spectral clustering is a leading and popular technique in unsupervised data analysis. Two of its major limitations are scalability and generalization of the spectral embedding (i.e., out-of-sample-extension). In this paper we introduce a deep learning approach to spectral clustering that overcomes the above shortcomings. Our network, which we call SpectralNet, learns a map that embeds input data points into the eigenspace of their associated graph Laplacian matrix and subsequently clusters them. We train SpectralNet using a procedure that involves constrained stochastic optimization. Stochastic optimization allows it to scale to large datasets, while the constraints, which are implemented using a specialpurpose output layer, allow us to keep the network output orthogonal. Moreover, the map learned by SpectralNet naturally generalizes the spectral embedding to unseen data points. To further improve the quality of the clustering, we replace the standard pairwise Gaussian affinities with affinities learned from the given unlabeled data using a Siamese network. Additional improvement of the resulting clustering can be achieved by applying the network to code representations produced, e.g., by standard autoencoders. Our end-to-end learning procedure is fully unsupervised. In addition, we apply VC dimension theory to derive a lower bound on the size of SpectralNet. State-of-the-art clustering results are reported on the Reuters dataset. Our implementation is publicly available at https://github.com/kstant0725/SpectralNet.

PAPER SUMMARY
This paper aims to address two limitations of spectral clustering: its scalability to large datasets and its generalizability to new samples. The proposed solution is based on designing a neural network called SpectralNet that maps the input data to the eigenspace of the graph Laplacian and finds an orthogonal basis for this eigenspace. The network is trained by alternating between orthogonalization and gradient descent steps, where scalability is achieved by using a stochastic optimization scheme that instead of computing an eigendecomposition of the entire data (as in vanilla spectral clustering) uses a Cholesky decomposition of the mini batch to orthogonalize the output. The method can also handle out-of-sample data by applying the learned embedding function to new data. Experiments on the MNIST handwritten digit database and the Reuters document database demonstrate the effectiveness of the proposed SpectralNet.
COMMENTS
1) I find that the output layer (i.e. the orthogonalization layer) is not well-justified. In principle, different batches require different weights on the output layer. Although the authors observe empirically that orthogonalization weights are roughly shared across different batches, the paper lacks a convincing argument for why this can happen. Moreover, it is not clear why an output layer designed to orthogonalized batches from the training set would also orthogonalize batches from the test set?
2) One claimed contribution of this work is that it extends spectral clustering to large scale data. However, the paper could have commented more on what makes spectral clustering not scalable, and how the method in this paper addresses that. The authors did mention that spectral clustering requires computing eigenvectors for large matrices, which is prohibitive. However, this argument is not entirely true, as eigen-decomposition for large sparse matrices can be carried out efficiently by tools such as ARPACK. On the other hand, computing the nearest neighbor affinity or Gaussian affinity is N^2 complexity, which could be the bottleneck of computation for spectral clustering on large scale data. But this issue can be addressed using approximate nearest neighbors obtained, e.g., via hashing. Overall, the paper compares only to vanilla spectral clustering, which is not representative of the state of the art. The paper should do an analysis of the computational complexity of the proposed method and compare it to the computational complexity of both vanilla as well as scalable spectral clustering methods to demonstrate that the proposed approach is more scalable than the state of the art.
3)  Continuing with the point above, an experimental comparison with prior work on large scale spectral clustering (see, e.g. [a] and the references therein) is missing. In particular, the result of spectral clustering on the Reuters database is not reported, but one could use other scalable versions of spectral clustering as a baseline.
4)  Another benefit of the proposed method is that it can handle out-of-sample data. However, the evaluation of such benefit in experiments is rather limited. In reporting the performance on out-of-sample data, there is no other baseline to compare with. One can at least compare with the following baseline: apply k-means to the training data in input space, and classify each test data to the nearest centroid.
5) The reason for using an autoencoder to extract features is unclear. In subspace clustering, it has been observed that features extracted from a scattering transform network [b] can significantly improve clustering performance, see e.g. [c] where all methods have >85% accuracy on MNIST. The methods in [c] are also tested on larger datasets.
[a] Choromanska, et. al., Fast Spectral Clustering via the Nystrom Method, International conference on algorithmic learning theory, 2013
[b] Bruna, Mallat, Invariant Scattering Convolution Networks, arXiv 2012
[c] You, et. al., Oracle Based Active Set Algorithm for Scalable Elastic Net Subspace Clustering, CVPR 2016


Published as a conference paper at ICLR 2018 SYNTHESIZING REALISTIC NEURAL POPULATION ACTIVITY PATTERNS USING GENERATIVE ADVERSARIAL NETWORKS
The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing. Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons. We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still benefiting from parameter sharing in the temporal domain. We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the first-and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics. We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-ofthe-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks. Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more flexible method than these alternative approaches. Finally, we show how to exploit a trained Spike-GAN to construct 'importance maps' to detect the most relevant statistical structures present in a spike train. Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.

[Summary of paper] The paper presents a method for simulating spike trains from populations of neurons which match empirically measured multi-neuron recordings. They set up a Wasserstein-GAN and train it on both synthetic and real multi-neuron recordings, using data from the salamander retina. They find that their method (Spike-GAN) can produce spike trains that visually look like the original data, and which have low-order statistics (firing rates, correlations, time-lagged-correlations, total sum of population activity) which matches those of the original data. They emphasize that their network architecture is 'semi-convolutional', i.e. convolutional in time but not across neurons. Finally, they suggest a way to analyse the fitted networks in order to gain insights into what the 'relevant' neural features are, and illustrate it on synthetic data into which they embedded these features.
[Originality] This paper falls into the category of papers that do a next obvious thing ("GANs have not been applied to population spike trains yet"), and which do it pretty well: If one wants to create simulated neural activity data which matches experimentally observed one, then this method indeed seems to do that. As far as I know, this would be the first peer-reviewed application of GANs to multi-neuron recordings of neural data (but see https://arxiv.org/abs/1707.04582 for an arxiv paper, not cited here-- should be discussed at least).  On a technical level, there is very little to no innovation here -- while the authors emphasise their 'semi-convolutional' network architecture, this is obviously the right architecture to use for multivariate time-series data, and in itself not a big technical novel. Therefore, the paper should really be evaluate as an `application' paper, and be assessed in terms of i) how important the application is, ii) how clearly it is presented, and iii) how convincing the results are relative to state of the art.
i) [Importance of problem, potential significance] Finding statistical models for modelling and simulating population spike trains is a topic which is extensively studied in computational neuroscience, predominantly using  model-based approaches using MaxEnt models, GLMs or latent variable models. These models are typically simple and restricted, and certainly fall short of capturing the full complexity of neural data. Thus, better, and more flexible solutions for this problem would certainly be very welcome, and have an immediate impact in this community.  However, I think that the approach based on GANs actually has two shortcomings which are not stated by the authors, and which possibly limit the impact of the method: First, statistical models of neural spike trains are often used to compute probabilities e.g. for decoding analyses— this is difficult or impossible for GANs. Second, one most often does not want to simulate data which match a specific recording, but rather which have specified statistics (e.g. firing rates and correlations)— the method here is based on fitting a particular data-set, and it is actually unclear to me when that will be useful.
ii) [Clarity] The methods are presented and explained clearly and cleanly. In my view, too much emphasis is given to highlighting the ‘semi-convolutional’ network, and, conversely, practical issues (exact architectures, cost of training) should be explained more clearly, possibly in an appendix. Similarly, the method would benefit from the authors releasing their code.
iii) [Quality, advance over previous methods] The authors discuss several methods for simulating spike trains in the introduction. In their empirical comparisons, however, they completely focus on a particular model-class (maximum entropy models, ME) which they label being the ‘state-of-the-art’. This label is misleading— ME models are but one of several approaches to modelling neural spike trains, with different models having different advantages and limitations (there is no benchmark which can be used to rank them...). In particular, the only ‘gain’ of the GAN over ME  models in the results comes from their ability of the GAN to match temporal statistics. Given that the ME models used by the authors are blind to temporal correlations, this is, of course (and as pointed out by the authors) hardly surprising. How does the GAN approach fair against alternative models which do take temporal statistics into account, e.g. GLMs, or simple moment-based method e.g. Krumin et al 2009, Lyamzin 2010, Gutnisky et al 2010— setting these up would be simple, and it would provide a non-trivial baseline for the ability of spike-GAN to outperform at least these models? While it true that GANs are much more expressive than the model-based approaches used in neuroscience, a clear demonstration would have been useful.
Minor comments:
  - p.3: The abbreviation “1D-DCGAN” is never spelled out.
  - p.3: The architecture of Spike-GAN is never explicitely given.
  - p.3: (Sec. 2.2) Statistic 2) “average time course across activity patterns” is unclear to me -- how does one select the activity patterns over which to average? Moreover, later figures do not seem to use this statistic.
  - p.4: “introduced correlations between randomly selected pairs” -- How many such pairs were formed?
  - p.7 (just above Discussion) At the beginning of this section, and for Figs. 4A,B, the texts suggests that packets fire spontaneously with a given probability. For Figs. 4C-E, a particular packet responds to a particular input. Is then the neuron population used in these figures different from the one in Figs. 4A,B? How did the authors ensure that a particular set of neurons respond to their stimulus as a packet? What stimulus did they use?
  - p.8 (Fig. 4E) Are the eight neurons with higher importance those corresponding to the packet? This is insinuated but not stated.
  - p.12 (Appendix A)
    + The authors do not mention how they produced their “ground truth” data. (What was its firing rate? Did it include correlations? A refractory period?)
    + Generating samples from the trained Spike-GAN is ostensibly cheap. Hence it is unclear why the authors did not  produce a large enough number of samples in order to obtain a 'numerical probability', just as they did for the ground truth data?
    + Fig. S1B: The figure shows that every sample has the same empirical frequency. This indicates more a lack of statistical power rather than any correspondence between the theoretical and empirical probabilities. This undermines the argument in the second paragraph of p.12. In the other hand, if the authors did approximate numerical probabilities for the Spike-GAN, this argument would no longer be required.
  - p.13 Fig. S1A,B: the abscissas mention “frequency”, while the ordinates mention “probability”
  - p.25 Fig. S4: This figure suggests that the first layer of the Spike-GAN critic sometimes recognizes the packet patterns in the data. However, to know whether this is true, we would need to compare this to a representation of the neurons reordered in the same way and identified by packet. I.e. one expects something something like figure like Fig. 4A, with the packets lining up with the recovered filters when neurons are ordered the same way.


We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks. With the proposed method, the label embedding is adaptively and automatically learned through back propagation. The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process. As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed. Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable. The proposed method achieves comparable or even better results than the state-of-the-art systems.

The paper proposes to add an embedding layer for labels that constrains normal classifiers in order to find label representations that are semantically consistent. The approach is then experimented on various image and text tasks.
The description of the model is laborious and hard to follow. Figure 1 helps but is only referred to at the end of the description (at the end of section 2.1), which instead explains each step without the big picture and loses the reader with confusing notation. For instance, it only became clear at the end of the section that E was learned.
One of the motivations behing the model is to force label representations to be in a semantic space (where two labels with similar meanings would be nearby). The assumption given in the introduction is that softmax would not yield such a representation, but nowhere in the paper this assumption is verified. I believe that using cross-entropy with softmax should also push semantically similar labels to be nearby in the weight space entering the softmax. This should at least be verified and compared appropriately.
Another motivation of the paper is that targets are given as 1s or 0s while soft targets should work better. I believe this is true, but there is a lot of prior work on these, such as adding a temperature to the softmax, or using distillation, etc. None of these are discussed appropriately in the paper.
Section 2.2 describes a way to compress the label embedding representation, but it is not clear if this is actually used in the experiments. h is never discussed after section 2.2.
Experiments on known datasets are interesting, but none of the results are competitive with current state-of-the-art results (SOTA), despite what is said in Appending D. For instance, one can find SOTA results for CIFAR100 around 16% and for CIFAR10 around 3%. Similarly, one can find SOTA results for IWSLT2015 around 28 BLEU. It can be fine to not be SOTA as long as it is acknowledged and discussed appropriately.


We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions. Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions. On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art. Furthermore, DRN generalizes the conventional multilayer perceptron (MLP). In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.

Summary:
This paper presents a new network architecture for learning a regression of probability distributions.
The distribution output from a given node is defined in terms of a learned conditional probability function, and the output distributions of its input nodes. The conditional probability function is an unnormalized distribution with the same form as the Boltzman distribution, and distributions are approximated from point estimates by discretizing the finite support into predefined equal-sized bins. By letting the conditional distribution between nodes be unnormalized, and using an energy function that incorporates child nodes independently, the approach admits efficient computation that does not need to model the interaction between the distributions output by nodes at a given level.
Under these dynamics and discretization, the chain rule can be used to derive a matrix of gradients at each node that denotes the derivative of the discretized output distribution with respect to the current node's discretized distribution. These gradients are in turn used to calculate updates for the network parameters with respect to the Jensen Shannon divergence between the predicted distribution and a target distribution.
The approach is evaluated on three tasks, two synthetic and one real world. The baselines are the state of the art triple basis estimator (3BE) or a standard MLP that represents the output distribution using a softmax over quantiles. On both of the synthetic tasks --- which involve predicting gaussians --- the proposed approach can fit the data reasonably using far fewer parameters than the baselines, although 3BE does achieve better overall performance. On a real world task that involves predicting a distribution of future stock market prices from multiple input stock marked distributions, the proposed approach significantly outperforms both baselines. However, this experiment uses 3BE outside of its intended use case --- which is for a single input distribution --- so it's not entirely clear how well the very simple proposed model is doing.
Notes to authors:
I'm not familiar with 3BE but the fact that it is used outside of its intended use case for the stock data is worrying. How does 3BE perform at predicting the FTSE distribution at time t + k from the FTSE distribution at time t only? Do the multiple input distributions actually help?
You use a kernel density estimate with a Gaussian kernel function to estimate the stock market pdf, but then you apply your network directly to this estimate. What would happen if you built more complex networks using the kernel values themselves as inputs?
Could you also run experiments on the real-world datasets used by the 3BE paper?
What is the structure of the DRN that uses > 10^3 parameters (from Fig. 4)? The width of the network is bounded by the two input distributions, so is this network just incredibly deep? Also, is it reasonable to assume that both the DRN and MLP are overfitting the toy task when they have access to an order of magnitude more parameters than datapoints.
It would be nice if section 2.4 was expanded to actually define the cost gradients for the network parameters, either in line or in an appendix.