The Large Margin Mechanism for Differentially Private Maximization
A basic problem in the design of privacy-preserving algorithms is the private maximization problem: the goal is to pick an item from a universe that (approximately) maximizes a data-dependent function, all under the constraint of differential privacy. This problem has been used as a sub-routine in many privacy-preserving algorithms for statistics and machine learning. Previous algorithms for this problem are either range-dependent-i.e., their utility diminishes with the size of the universe-or only apply to very restricted function classes. This work provides the first general purpose, range-independent algorithm for private maximization that guarantees approximate differential privacy. Its applicability is demonstrated on two fundamental tasks in data mining and machine learning.

This paper looks at differentially private algorithms for a generic maximization problem ("private argmax" might be a good name). Given a collection of K of items, and a data set D of n individuals, and a "score" function f that assigns each item i a data-based score f(i;D), the goal is to find an item i with approximately maximal score, while preserving differential privacy. This "private argmax" has proven to be a fundamental problem in the theory of private data analysis. It was first formulated by McSherry and Talwar (2007), who proposed the "exponential mechanism" to solve it. Because of its generality, that mechanism has been a central tool for designing many differentially private algorithms. The performance of that algorithm is optimal in the worst case, even among "approximately DP" algorithms, but it has been known for some time that there are various simple conditions under which it can be improved substantially.The authors give a private argmax algorithm which is differentially private (actually, a commonly-studied relaxation called "approximate DP") and which does better than the exponential mechanism for a large class of inputs. The authors call these instances "large margin" instances -- essentially, they assume that there are not too many items with near-optimal margin. Previously, it was known how to solve this problem when there is a large gap between the best and second-best items; this paper generalizes that technique considerably. The authors illustrate their results by showing theoretical improvements for a two problems. The more interesting of these is PAC learning -- the authors show that their algorithms allows one to exploit "shell bounds" (essentially, conditions which imply that not too many concept classes are good for the data set with high probability). This is a nice result that has the potential to influence future work considerably, as it provides a useful and basic algorithmic tool. It is not completely unexpected, given recent work of a similar flavor, but it takes those ideas much further. I recommend accepting the paper. The paper also presents lower bounds showing that the parameters of the algorithm cannot be asymptotically improved. The second of these (for (alpha,delta)-DP) is a nice contribution independently of the rest of the paper. Comments: I think the presentation of the algorithm could be somewhat simplified. - For any given ell, there is a simple mechanism which does well and is private *under the (ell,gamma) assumption* for gamma >= log(\ell/delta)/alpha. Namely, just run the exponential mechanism on the top \ell items. (In a nutshell, it is private since the probability that one outputs an item with frequency f_\ell is at most delta). This is somewhat simpler than the truncated exponential mechanism the authors employ. - One can select a good ell using the sparse vector technique. This is essentially what the authors do.Such a two-stage explanation might make the algorithm a little easier to parse.This paper would probably be more appealing to the NIPS community with some sort of empirical evaluation, but I think it is more than strong enough as it is for acceptance.


DEEP LEARNING GENERALIZES BECAUSE THE PARAMETER-FUNCTION MAP IS BIASED TOWARDS SIMPLE FUNCTIONS
Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime where classical learning theory would instead predict that they would severely overfit. While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit. In this paper, we provide a new explanation. By applying a very general probability-complexity bound recently derived from algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions. We then provide clear evidence for this strong bias in a model DNN for Boolean functions, as well as in much larger fully conected and convolutional networks trained on CIFAR10 and MNIST. As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems. This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than the more conventional prior over parameter space. If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets. By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood, we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10and for architectures including convolutional and fully connected networks.

The authors make a case that deep networks are biased
toward fitting data with simple functions.
The start by examining the priors on classifiers obtained by sampling
the weights of a neural network according to different distributions.  They do this
in two ways.  First, they examine properties of the distribution
on binary-valued functions on seven boolean inputs obtained by
sampling the weights of a small neural network.  They also empirically compare
the labelings obtained by sampling the weights of a network with
labelings obtained from a Gaussian process model arising from earlier
work.
Next, they analyze the complexity of the functions produced, using
different measures of the complexity of boolean functions.  A
favorite of theirs is something that they call Lempel-Ziv complexity,
which is measured by choosing an arbitrarily ordering of the
domain, writing the outputs of the function in that ordering,
and looking at how well the Lempel-Ziv algorithm compresses this
sequence.  I am not convinced that this is the most meaningful
and fundamental measure of the complexity of functions.
(In the supplementary material, they examine some others.
They show plots relating the different measures in the body
of the paper.  None of the measures is specified in detail in the
body of the paper. They provide plots relating these complexity
measures, but they don't demonstrate a very close connection.)
The authors then evaluate the generalization bound obtained by
applying a PAC Bayes bound, together with the assumption that
the training process produces weights sampled from the distribution
obtained by conditioning weights chosen according to the random
initialization on the event that they fit they fit the training
data perfectly.  They do this for small networks and simple datasets.
They bounds are loose, but not vacuous, and follow the same order
of difficulty on a handful of datasets as the true generalization
error.
In all of their experiments, they stop training when the training
accuracy reaches 100%, where papers like https://arxiv.org/pdf/1706.08947.pdf
have found that continuing training past this point further improves test
accuracy.  The experiments all use architectures that are
quite dissimilar to what is commonly used in practice, and
achieve much worse accuracy, so that a reader is concerned
that the results differ qualitatively in other respects.
I do not find it surprising that randomly sampling parameters
of deep networks leads to simple functions.
Papers like the Soudry, et al paper cited in this submission are
inconsistent with the assumption in the paper that SGD samples
parameters uniformly.
It is not clear to me how many hidden layers were used for the
results in Table 1 (is it four?).
I did find it interesting to see exactly how concentrated the
distribution of functions obtained in their 7-input experiment
was, and also found results on the agreement of the Gaussian process
models with the randomly sampled weight interesting, as far as they
went.  Overall, I am not sure that this paper provided enough
fundamental new insight to be published in ICLR.


LEARNABLE EMBEDDING SPACE FOR EFFICIENT NEURAL ARCHITECTURE COMPRESSION
We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.

================
Post-Rebuttal
================
I thank the authors for the larger amount of additional work they put into the rebuttal. Since the authors addressed my main concerns, i e. comparison to existing methods,  clarifications of the proposed approach, adding references to related work, I will  increase my score and suggest to accept the paper.

The paper describes a new neural architecture search strategy based on Bayesian optimization to find a compressed version of a teacher network. The main contribution of the paper is to learn an embedding that maps from a discrete encoding of an architecture to a continuous latent vector such that standard Bayesian optimization can be applied.
The new proposed method improves in terms of compressing the teacher network with just a small drop in accuracy upon an existing neural architecture search method based on reinforcement learning and random sampling.
Overall, the paper presents an interesting idea to use Bayesian optimization on high dimensional discrete problems such as neural architecture search. I think a particular strength of this methods is that the embedding is fairly general and can be combined with various recent advances in Bayesian optimization, such as, for instance, multi-fidelity modelling.
It also shows on some compression experiments superior performance to other state-of-the-art methods.
However, in its current state I do not think that the paper is read for acceptance:
- Since the problem is basically just a high dimensional, discrete optimization problem, the paper misses comparison to other existing Bayesian optimization methods such as TPE [1] / SMAC [2] that can also handle these kind of input spaces. Both of these methods have been applied to neural architecture search [3][4] before. Furthermore, since the method is highly related to NASBOT [5], it would be great to also see a comparison to it.
- I assume that in order to learn a good embedding, similar architectures need to be mapped to latent vector that are close in euclidean space, such that the Gaussian process kernel can model any correlation[7]. How do you make sure that the LSTM learns a meaningful embedding space? It is also a bit unclear why the performance f is not used directly instead of p(f|D). Using f instead of p(f|D) would probably also make continual training of the LSTM easier, since function values do not change.
- The experiment section misses some details:
  - Do the tables report mean performances or the performance of single runs? It would also be more convincing if the table contains error bars on the reported numbers.
  - How are the hyperparameters of the Gaussian process treated?

- The related work section misses some references to Lu et al.[6] and Gomez-Bombarelli et al.[7] which are highly related.
- What do you mean with the sentence  "works on BO for NAS can only tune feed-forward structures" in the related work section? There is no reason why other Bayesian optimization should not be able to also optimize recurrent architectures (see for instance Snoek et al.[8]).
- Section 3.3 is a bit confusing and to be honest I do not get the motivation for the usage of multiple kernels. Why do the first architectures biasing the LSTM? Since Bayesian optimization with expected improvement samples around the global optimum, should not later evaluated, well-performing architectures more present in the training dataset for the LSTM?
[1] Algorithms for Hyper-Parameter Optimization
    J. Bergstra and R. Bardenet and Y. Bengio and B. Kegl
    Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NIPS'11)
[2] Sequential Model-Based Optimization for General Algorithm Configuration
    F. Hutter and H. Hoos and K. Leyton-Brown
    Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION'11)
[3] Towards Automatically-Tuned Neural Networks
    H. Mendoza and A. Klein and M. Feurer and J. Springenberg and F. Hutter
    ICML 2016 AutoML Workshop
[4] Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures
    J. Bergstra and D. Yamins and D. Cox
    Proceedings of the 30th International Conference on Machine Learning (ICML'13)
[5] Neural Architecture Search with Bayesian Optimisation and Optimal Transport
    K. Kandasamy and W. Neiswanger and J. Schneider and B. P{\'{o}}czos and E. Xing
    abs/1802.07191
[6] Structured Variationally Auto-encoded Optimization
    X. Lu and J. Gonzalez and Z. Dai and N. Lawrence
    Proceedings of the 35th International Conference on Machine Learning
[7] Automatic chemical design using a data-driven continuous representation of molecules
    R. Gómez-Bombarelli and J. Wei and D. Duvenaud and J. Hernández-Lobato and B. Sánchez-Lengeling and D. Sheberla and J. Aguilera-Iparraguirre and T. Hirzel. and R. Adams and A. Aspuru-Guzik
    American Chemical Society Central Science
[8] Scalable {B}ayesian Optimization Using Deep Neural Networks
    J. Snoek and O. Rippel and K. Swersky and R. Kiros and N. Satish and N. Sundaram and M. Patwary and Prabhat and R. Adams
    Proceedings of the 32nd International Conference on Machine Learning (ICML'15)


Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control. Deep neural networks have been adopted for sensor fusion in a body of recent studies. Among these, the so-called netgated architecture was proposed, which has demonstrated improved performances over the conventional convolutional neural networks (CNN). In this paper, we address several limitations of the baseline negated architecture by proposing two further optimized architectures: a coarser-grained gated architecture employing (feature) group-level fusion weights and a two-stage gated architectures leveraging both the group-level and featurelevel fusion weights. Using driving mode prediction and human activity recognition datasets, we demonstrate the significant performance improvements brought by the proposed gated architectures and also their robustness in the presence of sensor noise and failures.

This paper tackles the problem of sensor fusion, where multiple (possibly differing) sensor modalities are available and neural network architectures are used to combine information from them to perform prediction tasks. The paper proposed modifications to a gated fusion network specifically: 1) Grouping sets of sensors and concatenating them before further processing, and 2) Performing multi-level fusion where early sensor data representations are concatenated to produce weightings additional to the those obtained from features concatenated at a later stage. Experimental results show that these architectures achieve performance gains from 2-6%, especially when sensors are noisy or missing.
Strengths
 + The architectures encourage fusion at multiple levels (especially the second one), which is a concept that has been successful across the deep learning literature

 + The paper looks at an interesting topic, especially related to looking at the effects of noise and missing sensors on the gating mechanisms.
 + The results show some positive performance gains, although see caveats below.
Weaknesses

 - The related work paragraph is extremely sparse. Fusion is an enormous field (see survey cited in this paper as well [1]), and I find the small choice of fusion results with a YouBot to be strange. A strong set of related work is necessary, focusing on those that are similar to the work. As an example spatiotemporal fusion (slow fusion [2]) bears some resemblence to this work but there are many others (e.g. [3,4] as a few examples).

   [1] Ramachandram, Dhanesh, and Graham W. Taylor. "Deep multimodal learning: A survey on recent advances and trends." IEEE Signal Processing Magazine 34.6 (2017): 96-108.
 	   Ramach
   [2] Karpathy, Andrej, et al. "Large-scale video classification with convolutional neural networks." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2014
   [3] Mees, Oier, Andreas Eitel, and Wolfram Burgard. "Choosing smartly: Adaptive multimodal fusion for object detection in changing environments." Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on. IEEE, 2016.
   [4] Kim, J., Koh, J., Kim, Y., Choi, J., Hwang, Y., & Choi, J. W. (2018). Robust Deep Multi-modal Learning Based on Gated Information Fusion Network. arXiv preprint arXiv:1807.06233.
 - The paper claims to provide a "deep understanding of the relationships between sensory inputs, fusion weights, network architecture, and resulting performance". I don't think it really achieves
    this with the small examples of weights for some simple situations.
 - It is very unclear whether the architectures have more or less parameters. At one point it is stated that the original architecture overfits and the new architecture has less parameters (Sec 2.2 and 3). But then it is stated for fairness the number of neurons is equalized (5.2), and later in that section that the new architectures have additional neurons. Which of these is accurate?
 - Related to the previous point, and possibly the biggest weakness, the experimental methodology makes it hard to tell if performance is actually improved. For example, it is not clear to me that the performance gains are not just a result of less overfitting (for whatever reason) of baselines and that the fixed number of epochs therefore results in stopping at a better performance. Please show training and validation curves so that we can see whether the epochs chosen for the baselines are not just chosen after overfitting (in which case early stopping will improve the performance). As another example, there are no variances shown in the bar graphs.
 - The examples with noise and failures are limited. For example, it is also not clear why an increase of noise in the RPM feature (Table 5) actually increases the weight of that group in the two-stage architecture. What does that mean? In general there isn't any principled method proposed for analyzing these situations.
Some minor comments/clarifications:
  - What is the difference between these gated networks and attentional mechanisms, e.g. alpha attention (see "Attention is all you need" paper)?
  - What is a principled method to decide on the groupings?
  - There are several typos throughout the paper
    * "in the presence of snesor" => "in the presence of sensor"
    * Throughout the paper: "Predication" => "Prediction"
    * "Likelihood of stucking the training"
  - Tensorflow is not a simulation environment
Overall, the paper proposes architectural changes to an existing method for fusion, and while positive results are demonstrated there are several issues in the experimental methodology that make it unclear where the benefits come from. Further, the paper lacks novelty as multi-level fusion has been explored significantly and the changes are rather minor. There is no principled method or concepts that drive the architectural changes, and while the authors claim a deeper investigation into the networks' effectiveness under noise and failures the actual analysis is too shallow.


SMALL NONLINEARITIES IN ACTIVATION FUNCTIONS CREATE BAD LOCAL MINIMA IN NEURAL NETWORKS
We investigate the loss surface of neural networks. We prove that even for onehidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima. We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum. Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks. We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.

The authors provide a clean and easily understood sufficient
condition for spurious local minima to exist in networks with
a hidden layer using ReLUs or leaky ReLUs.  This condition,
that there is not linear transformation with zero loss,
is satisfied for almost all inputs with more examples than
input variables.
The construction is elegant.  The mathematical writing in the paper,
especially describing the proof of Theorem 1, is very nice -- they
expose the main ideas effectively.
I do not know of another paper using a similar proof, but I have not
studied the proofs of the most closely related papers prior to doing
this review, so I have limited ability to vouch for this paper's
technical novelty.
The authors also show that networks using many other popular
activation functions have spurious local minima for a very
simple dataset.  All of these analysis are unified using a
simple, if technical, set of conditions on activation function.
Finally, the authors prove a somewhat technical theorem about
optima in deep linear networks, which generalizes some
earlier treatments of this topic, providing an checkable
condition for global minimality.
There is extensive discussion of related work.  I am not aware of
related work not covered by the authors.
In some cases, when the authors discuss previous work, they write as
if restriction to the realizable case is an assumption, when it seems
to me to be more of a constraint.  In other words, it seems harder to
prove the existence of spurious minima in the realizable case.
They seem to acknowledge this after their statement of their Theorem 2,
which also uses a realizable dataset.
Also, a few papers, including the Venturi, et al paper cited by
the authors, have analyzed whether spurious local minima exist
in subsets of the parameter space, including those likely to
be reached during training with different sorts of initializations.
In light of this work, the authors might want to tone down claims
about how their work shows that results about linear networks do
not generalize to the non-linear case.  In particular, to make
their construction work in the case of wide networks, they
need an overwhelming majority of the hidden units to be "dead",
which seems as it is unlikely to arise from training with
commonly used initializations.
Overall, I think that this paper makes an interesting and
non-obvious contribution on a hot topic.


DISJOINT MAPPING NETWORK FOR CROSS-MODAL MATCHING OF VOICES AND FACES
We propose a novel framework, called Disjoint Mapping Network (DIMNet), for cross-modal biometric matching, in particular of voices and faces. Different from the existing methods, DIMNet does not explicitly learn the joint relationship between the modalities. Instead, DIMNet learns a shared representation for different modalities by mapping them individually to their common covariates. These shared representations can then be used to find the correspondences between the modalities. We show empirically that DIMNet is able to achieve better performance than the current state-of-the-art methods, with the additional benefits of being conceptually simpler and less data-intensive. The code is made available at https://github.com/ydwen/DIMNet.

# Summary
The article proposes a deep learning-based approach aimed at matching face images to voice recordings belonging to the same person.
To this end, the authors use independently parametrized neural networks to map face images and audio recordings -- represented as spectrograms -- to embeddings of fixed and equal dimensionality. Key to the proposed approach, unlike related prior work, these modules are not directly trained on some particular form of the cross-modal matching task. Instead, the resulting embeddings are fed to a modality-agnostic, multiclass logistic regression classifier that aims to predict simple covariates such as gender, nationality or identity. The whole system is trained jointly to maximise the performance of these classifiers. Given that (face image, voice recording) pairs belonging to the same person must share equal for these covariates, the neural networks embedding face images and audio recordings are thus indirectly encouraged to map face images and voice recordings belonging to the same person to similar embeddings.
The article concludes with an exhaustive set of experiments using the VGGFace and VoxCeleb datasets that demonstrates improvements over prior work on the same set of tasks.
# Originality and significance
The article follows-up on recent work [1, 2], building on their original application, experimental setup and model architecture. The key innovation of the article, compared to the aforementioned papers, lies on the idea of learning face/voice embeddings to maximise their ability to predict covariates, rather than by explicitly trying to optimise an objective related to cross-modal matching. While the fact that these covariates are strongly associated to face images and audio recordings had already been discussed in [1, 2], the idea of actually using them to drive the learning process is novel in this particular task.
While the article does not present substantial, general-purpose methodological innovations in machine learning, I believe it constitutes a solid application of existing techniques. Empirically, the proposed covariate-driven architecture is demonstrated to lead to better performance in the (VGGFace, VoxCeleb) dataset in a comprehensive set of experiments. As a result, I believe the article might be of interest to practitioners interested in solving related cross-modal matching tasks.
# Clarity
The descriptions of the approach, related work and the different experiments carried out are written clearly and precisely. Overall, the paper is rather easy to read and is presented using a logical, easy-to-follow structure.
In my opinion, perhaps the only exception to that claim lies in Section 3.4. If possible, I believe the Seen-Heard and Unseen-Unheard scenarios should be introduced in order to make the article self-contained.
# Quality
The experimental section is rather exhaustive. Despite essentially consisting of a single dataset, it builds on [1, 2] and presents a solid study that rigorously accounts for many factors, such as potential confounding due to gender and/or nationality driving prediction performance in the test set.
Multiple variations of the cross-modal matching task are studied. While, in absolute terms, no approach seems to have satisfactory performance yet, the experimental results seem to indicate that the proposed approach outperforms prior work.
Given that the authors claimed to have run 5 repetitions of the experiment, I believe reporting some form of uncertainty estimates around the reported performance values would strengthen the results.
However, I believe that the success of the experimental results, more precisely, of the variants trained to predict the "covariate" identity, call into question the very premise of the article. Unlike gender or nationality, I believe that identity is not a "covariate" per se. In fact, as argued in Section 3.1, the prediction task for this covariate is not well-defined, as the set of identities in the training, validation and test sets are disjoint. In my opinion, this calls into question the hypothesis that what drives the improved performance is the fact that these models are trained to predict the covariates. Rather, I wonder if the advantages are instead a "fortunate" byproduct of the more efficient usage of the data during the training process, thanks to not requiring (face image, audio recording) pairs as input.
# Typos
Section 2.4
1) "... image.mGiven ..."
2) Cosine similarity written using absolute value |f| rather than L2-norm ||f||_{2}
3) "Here we are give a probe input ..."
# References
[1] Nagrani, Arsha, Samuel Albanie, and Andrew Zisserman. "Learnable PINs: Cross-Modal Embeddings for Person Identity." arXiv preprint arXiv:1805.00833 (2018).
[2] Nagrani, Arsha, Samuel Albanie, and Andrew Zisserman. "Seeing voices and hearing faces: Cross-modal biometric matching." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.


Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model. We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units, regardless of the architecture and the dataset. We also show how a simple adaptation of an existing scoring function allows us to select the best units to prune. Finally, we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the early stages of training.

This paper proposes a simple improvement to methods for unit pruning. After identifying a unit to remove (selected by the experimenter’s pruning heuristic of choice), the activation of that unit is approximately incorporated into the subsequent unit by “mean replacement”. The mean unit activation (computed on a small subset of the training set) is multiplied by each outgoing weight (or convolutional filter) and added to each corresponding bias instead. Experiments show this method is generally better than the typical method of zero-replacement before fine-tuning, though the advantage is smaller after several epochs of fine-tuning.
While I find this paper intriguing and applaud the extensive experimentation and documentation, I have some concerns as well:
	1. There are unanswered questions about how this method relates to existing work. It is not clear from the paper how the “mean replacement” method differs from the two most related works (Ye, 2018) and (Morcos, 2018), which propose variations on replacing units with constant values or mean activations, respectively. Also, why does the method in this paper seem to yield good results, while the related method (Morcos, 2018) yields “inferior performance”?
	2. The results are stated to only apply to networks “without batch normalization”. The reason seems intuitive: any change that can be merely rolled into the bias will be lost after normalization (depending perhaps on the ordering of normalization and the non-linearities). This leaves an annually decreasing fraction of networks to which this method is applicable, given the widespread use of batch norm.
	3. Critically, it’s difficult to compare this work against other pruning works given the lack of results reported in terms of final test error and the lack of the ubiquitous “error vs. %-pruned” plot.

Overall, this paper is lacking some clarity, may be limited in originality, may be helpful for some common networks and composable with other pruning methods (significance), but has a good quality evaluation (subject to the clarity issues). I’m rating this paper below the threshold given the limitations, but I’m willing to consider an upgrade to the score if these questions are addressed.
Other notes:
	4. What is your definition of a convolutional “pruning unit”? (From context, I’d presume it corresponds to an output activation map.)
	5. In Section 3.1:  replace “in practice, people …” with  something like “in practice, it is common to”.
	6. In Equation 3, is the absolute value of the pruning penalty used in the evaluation?
	7. In the footnote in Section 3.2, how many training samples are needed for a good approximation? How many are used in the experiments?
	8. There are a couple typos in Section 3.2: “replacing -the- these units with zeroes” and “each of these output*s*”.
	9. Presumably the “\Delta Loss after pruning” in Figures 2-6 is validation or test loss, not training loss? Is this the cross-entropy loss? Also, it would be much easier to compare to other papers if test accuracy were reported instead or in addition.
	10. In Figure 4, the cost to recover using fine-tuning seems to be only roughly 2% of the original training time. How much time is lost to the process of computing the average unit activation?
UPDATE: I've raised the score slightly to 5 after the rebuttals and revisions.


Goal-oriented learning has become a core concept in reinforcement learning (RL), extending the reward signal as a sole way to define tasks. However, as parameterizing value functions with goals increases the learning complexity, efficiently reusing past experience to update estimates towards several goals at once becomes desirable but usually requires independent updates per goal. Considering that a significant number of RL environments can support spatial coordinates as goals, such as on-screen location of the character in ATARI or SNES games, we propose a novel goal-oriented agent called Q-map that utilizes an autoencoder-like neural network to predict the minimum number of steps towards each coordinate in a single forward pass. This architecture is similar to Horde with parameter sharing and allows the agent to discover correlations between visual patterns and navigation. For example learning how to use a ladder in a game could be transferred to other ladders later. We show how this network can be efficiently trained with a 3D variant of Q-learning to update the estimates towards all goals at once. While the Q-map agent could be used for a wide range of applications, we propose a novel exploration mechanism in place of ε-greedy that relies on goal selection at a desired distance followed by several steps taken towards it, allowing long and coherent exploratory steps in the environment. We demonstrate the accuracy and generalization qualities of the Q-map agent on a grid-world environment and then demonstrate the efficiency of the proposed exploration mechanism on the notoriously difficult Montezuma's Revenge and Super Mario All-Stars games.

Authors propose to overcome the sparse reward problem using an exploration strategy that incentivizes the agent to visit different parts of the game screen. This is done by building Q-maps, a 3D tensor that measures the value of the agent's current state (defined as the position of the agent) and action in reaching other (x, y) locations in the map. Each 2D slice of the Q-map measures the value at different (x, y) locations for one action. Such 2D slices (i.e. channels) are stacked together to form the Q-map. Taking the max across the channels, thus, provides the Q-value for the optimal action.
A policy for maximizing the rewards is trained using DQN. The Q-map based exploration is used as a replacement for \epsilon-greedy exploration.
The Q-map is used for exploration in the following way:
(a) Chose a random action with probability \epsilon_r.
(b) If neither a random action nor a "goal" is chosen, a new goal is chosen with probability \epislon_g. The goal is a (x, y) location, chosen so that is not too hard or too easy to reach it (i.e. Q-map values are neither too high or low; intuitively [1 - Q-map(x, y, a)] (for normalized/clipped Q) is a measure of distance of the goal).
     -- If a "goal" is chosen, the greedy action to go towards the goal is chosen.
(c) If neither a goal or random action is chosen, DQN is used to chose the greedy exploration.
Authors also bias the goal selection to match DQN's greedy action. This is done as following -- from a set of goals that satisfy (b) above; chose the goal for which Q-map selected action matches the DQN's greedy action.
Results are presented on simple 2D maze environments, Mario and Montezuma's revenge.
I have multiple concerns with the papers:
(i) The writing is informal and the ideas are not well explained. It would really benefit -- if authors introduce an algorithm box or talk about the method as a sequence of points. Right now, the ideas are scattered throughout the paper. I am still confused by figure 3 -- when are random goals chosen? Do random goals correspond to (b) above? Also, when the Horde architecture, GVF and UVF are mentioned, the references are missing -- I would love for the authors to include the corresponding  references.
(ii) The idea of reaching as many states as possible has been explored in count based visitation (Bellemare et al, Tang et al) — but no comparisons have been made to any previous work. Its always good to put a new work in the perspective of old work with similar ideas.
(iii) The authors propose biased and random goal sampling — I would love to see how much improvement does biased goal sampling offer over random goal sampling.
(iv) “…compare the performance of our proposed agent and a baseline DQN with a similar proportion of exploratory actions” .. I don’t agree with this a metric — I think the total number of steps is a good metric. Exploration is part of the agent’s algorithm to find the goal, we shouldn’t compare against DQN by matching the number of exploratory actions.
(v) “The Q-map is trained with transitions generated by randomly starting from any free locations in the environment and taking a random action.” Does this mean that when the agent is trained with Mario — the game is reset after every episode and the agent is placed a random starting location? If yes, then this is not a realistic assumption.
(vi) I would like to see — how do Q-maps generalize across levels of Mario or Montezuma’s revenge? Does Q-map trained on level-1 help in good exploration on future levels without any further fine-tuning?
Overall, I like the idea of incentivizing exploration without changing the reward function as is done in multiple prior works. However, I think more thorough quantitative evaluation is required and it will be interesting to see transfer of Q-maps outside the 2D-domains. I am happy to increase my score if such evidence is provided.
Other references worth including:
(a) Strategies for goal generation: Automatic Goal Generation for Reinforcement Learning Agents (https://arxiv.org/abs/1705.06366)


Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way. These methods have focussed largely on modifying the variational cost function to achieve this goal. However, we show that methods like β-VAE simplify the tendency of variational inference to underfit causing pathological overpruning and over-orthogonalization of learned components. In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered. Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space. This means there is no pressure to identify each true factor of variation with a latent variable. We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry. Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like β-VAE and TCVAE between reconstruction loss and disentanglement. The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

This paper presents a methodology to bring together independent subspace analysis and variational auto-encoders. Naturally, in order to do that, the authors propose a specific family of prior distributions that lead to subspace independence the Lp-nested distribution family. This prior distribution is then used to learn disentangled and interpretable representations. The mutual information gap is taken as the measure of disentanglement, while the reconstruction loss measures the quality of the representation. Experiments on the sPrites dataset are reported, and comparison with the state of the art shows some interesting results.
I understand the limitations of current approaches for learning disentangled representations, and therefore agree with the motivation of the manuscript, and in particular the choice of the prior distribution. However, I did not find the answer to some important questions, and generally speaking I believe that the contribution is not completely and clearly described.
P1) What is the shape of the posterior distribution?
P2) How does the reparametrization trick work in your case?
P3) How can one choose the layout of the subspaces, or this is also learned?
Moreover, and this is crucial, the proposed method is not clearly explained. Different concepts are discussed, but there is no summary and discussion of the proposed method as a whole. The reader must infer how the method works from the different pieces.
When discussing the performance of different methods, and even if in the text the four different alternatives are clearly explained, in figure captions and legens the terminology changes (ISA-VAE, ISA-beta-VAE, beta-VAE, beta-ISA-VAE, etc). This makes the discussion very difficult to follow, as we do not understand which figures are comparable to which, and in which respect.
In addition, there are other (secondary) questions that require an answer.
S1) After (10) you mention the subspaces v_1,...v_l_o. What is the formal definition of these subspaces?
S2) The definition of the distribution associated to ISA also implies that n_i,k = 1 for all i and k, right?
S3) Could you please formally write the family of distributions, since applying this to a VAE is the main contribution of your manuscript?
S4) Which parameters of this family are learned, and which of them are set in advance?
S5) From Figure 4 and 5, I understand that the distributions used are of the type in (7) and not (10). Can you comment on this?
S6) How is the Lp layout chosen?
S7) Why the Lp layout for ISA-beta-VAE in Figure 5 is not the same as in Figure 4 for ISA-VAE?
S8) What are the plots in Figure 4? They are difficult to interpret and not very well discussed.
Finally, there are a number of minor corrections to be made.
Abstract: latenT
Equation (3) missig a sum over j
Figure 1 has no caption
In (8), should be f(z) and not x.
Before (10), I understand you mean Lp-nested
I did not find any reference to Figure 3
In 4.1, the standard prior and the proposed prior should be referred to with different notations.
For all these reasons I recommend to reject the paper, since in my opinion it is not mature enough for publication.


LEARNING PROTEIN STRUCTURE WITH A DIFFERENTIABLE SIMULATOR
The Boltzmann distribution is a natural model for many systems, from brains to materials and biomolecules, but is often of limited utility for fitting data because Monte Carlo algorithms are unable to simulate it in available time. This gap between the expressive capabilities and sampling practicalities of energy-based models is exemplified by the protein folding problem, since energy landscapes underlie contemporary knowledge of protein biophysics but computer simulations are challenged to fold all but the smallest proteins from first principles. In this work we aim to bridge the gap between the expressive capacity of energy functions and the practical capabilities of their simulators by using an unrolled Monte Carlo simulation as a model for data. We compose a neural energy function with a novel and efficient simulator based on Langevin dynamics to build an end-toend-differentiable model of atomic protein structure given amino acid sequence information. We introduce techniques for stabilizing backpropagation under long roll-outs and demonstrate the model's capacity to make multimodal predictions and to, in some cases, generalize to unobserved protein fold types when trained on a large corpus of protein structures.

The paper proposes a new end-to-end training framework for computational prediction of protein structure from sequence.
This is a very important problem and any progress due to new data and/or methods for utilizing may have high impact.
The paper presents several technical contributions in the modelling and training procedure - for example, automatic transformation between Cartesian and angular coordinates, using Langevin dynamics, and imputation method to get fine atomic coordinates.
The overall breadth and depth of the methods presented in the paper are impressive. The paper describes a quite complicated systems with multiple modules interacting between them. The paper doesn't describe the system built in enough details, although many of the details are given in the appendix.
Figure. 6 presents a scheme of the entire system, but it lacks details about the different modules, and it is not clear how they interact and how their training together is performed.
The pseudo-code boxes describing Algorithms 1-4, and Table 2 describing the representation are informative and helpful, and more descriptions of this type would help.
For example:    - In Algorithm 3, what do 'CartesianStep' and 'ClippedInternalStep' mean? where are they described? (should have their own boxes/description).
		- I didn't see an Algorithm describing the atomic imputation part.
		- It would be good to add a high-level pseudo-code for the entire end-to-end training algorithm. In it there could be calls to Algorithms 1-4 when needed.
There is also no single place where all the parameters used by the authors to achieve their empirical results are presented
(e.g. learning rates, Gaussian kernel widths, how are random time steps for enforcing Lipschitz condition chosen etc.).
In addition, the empirical loss defined in eq. (8) is a sum of 6 different losses. It is not clear how are these very different losses scaled to the same 'units', which ones are more important,
if and how are constants multiplying them chosen to give lower/higher weights to some of the losses etc. - I guess these choices will have a large effect on the training.
The authors present generalization results of their trained model in predicting 3D structures from CATH at different generalization level
(i.e. different similarity levels to the training set proteins). It is not clear to me how good are these results, except that they are shown
to be better than a baseline simple model. How well does the author's model compare to other recently suggested end-to-end models?
(the authors mention AlQuraishi, Anand&Huang, papers). How do they compare to state-of-the art structure prediction programs? (e.g. CASP winners)?
I realize giving an automatic end-to-end solution is interesting even if performance is below that of best programs, but still it would be good to know gaps.
If such comparisons are less meaningful/not practical to perform this should be argued convincingly.
It would also be useful to add some metrics of running time - it is not clear how computationally heavy and scalable is the author's model and training, compared to other methods.
There are many typos and inconsistent notations which makes it harder for the reader to understand the paper.
For example, 'Figure ??' in multiple locations, wrong Figure referenced, using s vs. S for sequence - S is defined as an L*20 matrix but in the appendix there are
3 indices: s_{i,l,j} and it looks like different sequences in alignment should be denoted s_i.
Equation for M_{l,j} isn't clear: j is used both as fixed index and index in summation.
The indexing in 'orientation vectors' v-hat_ij definition seems off (the formula of base vectors gives 0/0)


We propose a non-adversarial feature matching-based approach to train generative models. Our approach, Generative Feature Matching Networks (GFMN), leverages pretrained neural networks such as autoencoders and ConvNet classifiers to perform feature extraction. We perform an extensive number of experiments with different challenging datasets, including ImageNet. Our experimental results demonstrate that, due to the expressiveness of the features from pretrained ImageNet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art results for challenging benchmarks such as CIFAR10 and STL10.

This paper proposes to learn implicit generative models by a feature matching objective which forces the generator to produce samples that match the means of the data distribution in some fixed feature space, focusing on image generation and feature spaces given by pre-trained image classifiers.
On the positive side, the paper is well-written and easy to follow, the experiments are clearly described, and the evaluation shows the method can achieve good results on a few datasets. The method is nice in that, unlike GANs and the stability issues that come with them, it minimizes a single loss and requires only a single module, the generator.
On the other hand, the general applicability of the method is unclear, the novelty is somewhat limited, and the evaluation is missing a few important baselines. In detail:
1) The proposed objective was used as a GAN auxiliary objective in [Salimans et al., 2016] and further explored in [Warde-Farley & Bengio, 2017]. The novel bit here is that the proposed objective doesn’t include the standard GAN term (so no need for an adversarially-optimized discriminator), and the feature extractor is a fixed pre-trained classifier or encoder from an auto-encoder (rather than a discriminator).
2) The method only forces the generator’s sample distribution to match the first moment (the mean) of the data distribution. While the paper shows that this can result in a generator that produces reasonably good samples in practice, it seems like this may have happened due to a “lucky” artifact of the chosen pre-trained feature extractors. For example, a degenerate generator that produces a single image whose features exactly match the mean would be a global optimum under this objective, equally good as a generator that exactly matches the data distribution. Perhaps no such image exists for the chosen pre-trained classifiers, but it’s nonetheless concerning that the objective does nothing to prevent this type of behavior in the general case. (This is similar to the mode collapse problem that often occurs with GAN training in practice, but at least a GAN generator is required to exactly match the full data distribution to achieve the global optimum of that objective.)
3) It’s unclear why the proposed ADAM-based Moving Average (AMA) updates are appropriate for estimate the mean features of the data distribution. Namely, unlike EMA updates, it’s not clear that this is an unbiased estimator (I suspect it’s not); i.e. that the expectation of the resulting estimates is actually the true mean of the dataset features.  It’s therefore not clear whether the stated objective is actually what’s being optimized when these AMA updates are used.
4) Related to (3), an important baseline which is not discussed is the true fixed mean of the dataset distribution. In Sec. 2.4 (on AMA) it’s claimed that “one would need large mini-batches for generating a good estimate of the mean features...this can easily result in memory issues”, but this is not true: one could trivially compute the full exact dataset mean of these fixed features by accumulating a sum over the dataset (e.g., one image a time, with minibatch size 1) and then dividing the result by the number of images in the dataset. Without this baseline, I can’t rule out that the method only works due to its reliance on the stochasticity of the dataset mean estimates to avoid the behavior described in (2), or even the fact that the estimates are biased due to the use of ADAM as described in (3).
5) The best results in Table 3 rely on initializing G with the weights of a decoder pretrained for autoencoding. However, the performance of the decoder itself with no additional training from the GFMN objective is not reported, so it’s possible that most of the result relies on *autoencoder* training rather than feature matching to get a good generator. This explanation seems especially plausible due to the fact that the learning rate is set to a miniscule value (5*10^-6 for ADAM, 1-2 orders of magnitude smaller than typical values). Without the generator pretraining, the next best CIFAR result is an Inception Score of 7.67, lower than the unsupervised result from [Warde-Farley & Bengio, 2017] of 7.72.
6) It is misleading to call the results based on ImageNet-pretrained models “unconditional” -- there is plenty of overlap in the supervision provided by the labeled images of the much larger ImageNet to CIFAR and other datasets explored here. This is especially true given that the reported metrics (Inception Score and FID) are themselves based on ImageNet-pretrained classifiers. If the results were instead compared to prior work on conditional generation (e.g. ProGAN (Karras et al., 2017), which reports CIFAR IS of 8.56), there would be a clear gap between these results and the state of the art.
Overall, the current version of the paper needs additional experiments and clarifying discussion to address these issues.
=======================================
REVISION
Based on the authors' responses, I withdraw points 3-5 from my original review. Thanks to the authors for the additional experiments. On (3), I indeed misunderstood where the moving average was being applied; thanks for the correction. On (4), the additional experiment using the global mean features for real data convinces me that the method does not rely on the stochasticity of the estimates. (Though, given that the global mean works just as well, it seems like it would be more efficient and arguably cleaner to simply have that be the main method. But this isn't a major issue.) On (5), I misread the learning rate specified for "using the autoencoder features" as being the learning rate for autoencoder *pretraining*; thanks for the correction. The added results in Appendix 11 do show that the pretrained decoder on its own does not produce good samples.
My biggest remaining concerns are with points (2) and (6) from my original review.
On (2), I did realize that features from multiple layers are used, but this doesn't theoretically prevent the generator from achieving the global minimum of the objective by producing a single image whose features are the mean of the features in the dataset. That being said, the paper shows that this doesn't tend to happen in practice with existing classifiers, which is an interesting empirical contribution. (It would be nice to also see ablation studies on this point, showing the results of training against features from single layers across the network.)
On (6), I'm still unconvinced that making use of ImageNet classifiers isn't providing something like a conditional training signal, and that using such classifiers isn't a bit of an "unfair advantage" vs. other methods when the metrics themselves are based on an ImageNet classifier. I realize that ImageNet and CIFAR have different label sets, but most if not all of the CIFAR classes are nonetheless represented -- in a finer-grained way -- in ImageNet. If ImageNet and CIFAR were really completely unrelated, an ImageNet classifier could not be used as an evaluation metric for CIFAR generators. (And yes, I saw the CelebA results, but for this dataset there's no quantitative comparison with prior work, and qualitatively, if the results are as good as or better than the 3 year old DCGAN results, I can't tell.)
On the other hand, given that the approach relies on these classifiers, I don't have a good suggestion for how to control for this and make the comparison with prior work completely fair. Still, it would be nice to see acknowledgment and discussion of this caveat in a future revision of the paper.
Overall, given that most of my concerns have been addressed with additional experiments and clarification, and that the paper is well-written and has some interesting results from its relatively simple approach, I've raised my rating to above acceptance threshold.


TREE-STRUCTURED RECURRENT SWITCHING LINEAR DYNAMICAL SYSTEMS FOR MULTI-SCALE MODELING
Many real-world systems studied are governed by complex, nonlinear dynamics. By modeling these dynamics, we can gain insight into how these systems work, make predictions about how they will behave, and develop strategies for controlling them. While there are many methods for modeling nonlinear dynamical systems, existing techniques face a trade off between offering interpretable descriptions and making accurate predictions. Here, we develop a class of models that aims to achieve both simultaneously, smoothly interpolating between simple descriptions and more complex, yet also more accurate models 1 . Our probabilistic model achieves this multi-scale property through a hierarchy of locally linear dynamics that jointly approximate global nonlinear dynamics. We call it the tree-structured recurrent switching linear dynamical system. To fit this model, we present a fully-Bayesian sampling procedure using Pólya-Gamma data augmentation to allow for fast and conjugate Gibbs sampling. Through a variety of synthetic and real examples, we show how these models outperform existing methods in both interpretability and predictive capability.

PAPER SUMMARY:
This paper introduces a probabilistic generative framework to model linear dynamical systems at multiple levels of resolution, where the entire complex, nonlinear dynamics is approximated via a hierarchy of local regimes of linear dynamics -- the global dynamic is then characterized as a switching process that switches between linear regimes in the hierarchy.
NOVELTY & SIGNIFICANCE:
The key contributions of this paper are (a) the use of tree-structured stick breaking to partition the entire dynamic space into a hierarchy of linear regimes; (b) the design of a hierarchical prior that is compatible to the tree structure; and (c) the developed Bayesian inference framework for it in Section 4.
By exploiting the tree-structured stick breaking process (Adams et al., 2010), the proposed framework is able to partition the entire dynamic space into a hierarchy of switching linear regimes.
This allows the dynamic to be queried at multiple levels of resolution. This appears to be the key difference between the proposed framework and the previous work of (Linderman et al., 2017) on recurrent switching dynamical systems that partition the dynamic space sequentially at the same level of resolution.
This seems like a non-trivial extension to the previous work of (Linderman et al., 2017) & I tend to consider this a novel contribution. That said, the paper was also not positioned against existing literature on hierarchical switching linear dynamic systems (see below) & I find it hard to evaluate the significance of the proposed framework (which explains the borderline rating)
"A Hierarchical Switching Linear Dynamical System applied to the detection of sepsis in neonatal condition monitoring", Ioan Stanculescu, Christopher K. I. Williams and Yvonne Freer. In Proceeding of the 30th Conference on Uncertainty in AI (UAI-14), pages 752-761
Could the authors please discuss the differences between the proposed work & (at least) the above?
TECHNICAL SOUNDNESS:
The technical exposition makes sense to me. Please also discuss the processing complexity of the resulting TrSLDS framework. In exchange for the improved performance, how much slower TrSLDS is as compared
to rSLDS? I am interested to see this demonstrated in the empirical studies.
CLARITY:
The paper is clearly written.
EMPIRICAL RESULTS:
The experiments look interesting and are very extensive on both test domains. However, I do not understand why the authors decided not to compare with rSLDS using its benchmark?
I find this somewhat sloppy and hope the authors would clarify this too.
****
Post-rebuttal update: The authors have made significant revision to their work, which sufficiently addressed all my concerns. I have upgraded my score accordingly and I am willing to support the acceptance of this paper.


One of the key challenges of session-based recommender systems is to enhance users' purchase intentions. In this paper, we formulate the sequential interactions between user sessions and a recommender agent as a Markov Decision Process (MDP). In practice, the purchase reward is delayed and sparse, and may be buried by clicks, making it an impoverished signal for policy learning. Inspired by the prediction error minimization (PEM) and embodied cognition, we propose a simple architecture to augment reward, namely Imagination Reconstruction Network (IRN). Specifically, IRN enables the agent to explore its environment and learn predictive representations via three key components. The imagination core generates predicted trajectories, i.e., imagined items that users may purchase. The trajectory manager controls the granularity of imagined trajectories using the planning strategies, which balances the long-term rewards and short-term rewards. To optimize the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of simulated trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using model-free algorithms. Empirically, IRN promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance compared to several baselines. Somewhat surprisingly, IRN using only the purchase reward achieves excellent next-click prediction performance, demonstrating that the agent can "guess what you like" via internal planning.

The paper aimed at improving the performance of recommendation systems via reinforcement learning. The author proposed an Imagination Reconstruction Network for the recommendation task, which implements an imagination-augmented policy via three components: (1)  the imagination core (IC) that predicts the next time steps conditioned on actions sampled from an imagination policy; (2) the trajectory manager (TM) that determines how to roll out the IC under the planning strategy and produces a set of imagined item trajectories; (3) the imagination-augmented executor (IAE) that aggregates the internal data resulting from imagination and external rewarding data to update its action policy.
Strengths of the paper:
(1) The research problem that the performance of recommendation systems needs to be improved is of great value to be investigated, as recommendation systems play crucial role in people’s daily lives.
(2) Experiments were conducted on a publicly available dataset.
(3) Robustness to cold-start scenario was tested and evaluated in the experiments.
Weaknesses of the paper:
(1) The motivations of applying reinforcement learning techniques are not convinced to me. There are a lot of supervised learning algorithms to the task of recommendations. Why do the authors utilize reinforcement learning to the task but not other supervised learning techniques? Is it because reinforcement learning based methods work better than traditional machine learning based ones? The motivations of integrating A3C (Asynchronous Advantage Actor-Critic) but not other techniques into the proposed model are not convinced to me as well.
(2) State-of-the-art reinforcement learning algorithms were not taken into account for baselines in the experiments. As the proposed method is built based on reinforcement learning, it would be better if the authors could include state-of-the-art reinforcement learning algorithms as their baselines.
(3) Some details are missing, resulting in the fact that it is hard for other researchers to fully capture the mechanism of the proposed algorithm. In equations (2) and (3), what is theta_v? How is theta_v associated with the parameters in LSTM. Is theta_v denoted the parameters of LSTM? How do the authors define the loss functions, i.e., \mathcal{L}_{A3C} and \mathcal{L}_{IRN}? What are the relationships among \mathcal{L}_{A3C}, \mathcal{L}_{IRN} and the one defined in equation (4)?
(4) The contributions of the paper in terms of theory are somewhat not significant. It seems that the proposed algorithm is built based on and combined by existing algorithms such as A3C.
Minor comments:
(1) It would be better if the authors can test the proposed model on more datasets. There are many publicly available datasets for testing the performance of recommendation systems.
(2) Figure 2 is not straightforward. It would be better if the authors can draw the figure in other ways. (I am not sure if the authors have expressed the underlying ideas clearly with Figure 2).


JANOSSY POOLING: LEARNING DEEP PERMUTATION- INVARIANT FUNCTIONS FOR VARIABLE-SIZE INPUTS
We consider a simple and overarching representation for permutation-invariant functions of sequences (or multiset functions). Our approach, which we call Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence. This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and flexible permutation-invariant functions. If carried out naively, Janossy pooling can be computationally prohibitive. To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with k-order interactions, and stochastic optimization algorithms with random permutations. Our framework unifies a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions. We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.

I really enjoyed this paper. It takes an idea which at first glance seems to be obviously bad (if you want permutation invariance, build a model that considers all permutations) and uses it to make the important point that the universal approximation results contained in Deep Sets [Zaheer et al. 2017] are not the last word on pooling. Janossy Pooling is intractable for most problems of interest (because it sums over all n! permutations of the input set) so the authors suggest 3 tractable alternatives: canonical orderings, k-ary dependencies and SGD / sampling-based approaches. Only the latter two are explored in detail, so I’ll focus on them:
K-ary dependencies
Functions that are restricted to k-ary dependencies in Janossy Pooling require summing over only |h|! / (|h| - k)! terms - that is they sum over the permutations of subsets of h of length k. In the experimental section, the authors show that this recovers some of the performances lost by using sum / mean pooling (as in Deep Sets), but this suggests the natural question: is it the fact that you’re explicitly modelling higher-order interactions that improves performance? Or is it that you’re doing Janossy pooling over the higher order interactions (i.e. summing over permutations of non-invariant functions)?
These two effects could be separated by comparing to invariant models that allow higher order interactions. E.g. you could compare against Santoro et al. [2017] who explicitly model pairwise interactions (or similarly any of the graph convolutional models [Kipf and Welling 2016, Hamilton et al 2017, etc.] with a fully connected graph would do the same); similarly Hartford, et al. [2018] allow for k-wise interactions by extending Deep Sets to exchangeable tensors - the permutation invariant analog of k-ary Janossy Pooling. All of these approaches model k-wise interactions through sum-pooling over permutation invariant functions so this lets you address the question - is it the permutation invariance that’s the problem (necessitating k-ary Janossy pooling) or is it the lack of higher-order interaction terms?
SGD approaches:
I think that the point that the sampling-based approaches are bias with respect to the Janossy sum is important to make and I liked the discussion around it, but I don’t follow the relevance of Proposition 2.2? I see that it gives conditions under which we can expect \pi-SGD to converge, but we aren’t provided with any guidance about how likely those conditions are to be satisfied? Furthermore - these conditions don’t seem to be specific to \pi-SGD - any SGD algorithm with “slightly biased” gradients that satisfy these conditions would converge. The regularization idea is interesting, but it isn’t evaluated so we’re left with theory that doesn’t provide guidance and isn’t evaluated.
Summary:
There are two ways to read this paper:
 1. Janossy pooling as a framework & proposed pooling approach implemented in one of the two ways discussed above.
 2. Janossy pooling as an intractable upper bound on what we might want from a pooling method (with approximations in the form of the LSTM approaches) and a demonstration that our current invariant pooling methods are insufficient.
I liked the paper based on reading (2). Janossy pooling clearly demonstrates limitations to sum / mean pooling which is widely used in practice which shows the need for better alternatives and it is on this basis that I’m arguing for it’s acceptance. My view is that the experimental section is too limited to support reading (1) which asserts that k-ary pooling or LSTM + sampling approaches are the right solution to this problem.
[Zaheer et al. 2017] - Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and
Alexander Smola. Deep Sets
[Santoro et al. 2017] - Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning.
[Kipf and Welling 2016] - Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Net- works
[Hamilton et al 2017] - William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs
[Hartford, et al. 2018] - Jason S. Hartford, Devon R. Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh. Deep models of interactions across sets


Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system. We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction.

The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset. Their analysis shows what kind of hidden state dynamics the GRU can approximate in one and two dimensions. In the experimental part, they show how a GRU with two hidden states trained on a multistep prediction task can learn such dynamics.
Although RNNs are important for Machine Learning, the paper seems to contain flaws in the theoretical part, which seem to invalidate some of the claimed results. But we may change our rating in case of a convincing rebuttal.
The Proof of Lemma 2 claims that h(t) achieves all values on the real set, which is false (h(t) assumes values in (-1,1)). Nevertheless, the theorem should hold since there is always at least one intersection between h and tanh(f(h)).
Lemma 1 claims that for any choice of parameters, there exist only finitely many fixed points. However, in the proof the authors only show that the number of fixed points cannot be uncountable, without taking into consideration the possibility that there are countably many fixed points. The proof also omits steps concerning the Taylor expansion which would make the proof clearer: We suggest adding those steps in the appendix. Furthermore, when equation (12) is Taylor-expanded, the authors do not consider the case where the GRU parameters are such that the argument of function “sech” is outside its convergence radius. These might be parameters for which there are infinitely many fixed points, even if we are unable to provide a Taylor expansion. The Lemma may still be correct, but this does not seem to be a complete proof.
The authors claim that an arbitrarily close approximation of a line attractor can be created using two GRUs, but no proof is provided.
The experimental part is difficult to evaluate since there are no learning curves for the three tasks. For instance, it is difficult to judge whether the GRUs are unable to learn the dynamics of a ring attractor because of theoretical limitations or because the model has not been properly trained for the specific task.
The paper is easy to read, except for certain parts where it is not clear if some of the statements are true in general or just have not been proven false by the authors. It is not clear why Figure 3 is representing all possible simple fixed points and bifurcation fixed points: is there a theoretical result stating that these are the only possible topologies, or are these the only ones found? The same question applies for the 36 images in figure 9. The range of the parameters used for finding these configurations is not specified.
Since the hidden state assumes values in (-1,1)^2, why is its range in most of the images (-1.5,1.5)?
We are not familiar with related work on transformations from discrete to continuous dynamical systems: are the dynamics of the discrete time GRU model preserved in the transformation? If so, is there a reference for this? Are the phase portraits in the middle row of figure 8 generated by letting the discrete GRU system evolve, or is the continuous system used with the parameters of the trained GRU?
We would like to see more explanations of why various topologies are useful for the applications mentioned in the paper. Given a generic dataset, how can these results help to understand how well a GRU will perform?
What is the reason behind the belief that the analysis extends to higher dimensions? The effects of a 1D -> 2D extension are far from trivial - why should that be different for higher dimensions?
The problem the authors want to solve seems important, and some of the theoretical results are promising, but we think that this paper has to be further polished before acceptance.
It is possible that we will increase the score if the authors can provide clarifications on the above questions.
Additional comments:
Introduction
-The vanishing gradient problem was not discovered in 1994, but in 1991 by Hochreiter:
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich, 1991. Advisor J. Schmidhuber.
- Make clear that GRU is a variant of vanilla LSTM with forget gates (where one gate is missing):
Gers et al. “Learning to Forget: Continual Prediction with LSTM.“ Neural Computation, 12(10):2451-2471, 2000.
- The intro says that GRU has become widely popular and cites Britz et al 2017, but Britz et al actually show that LSTM consistently outperforms its variant GRU in Neural Machine Translation. Please clarify this.
- Also mention Weiss et al (“On the Practical Computational Power of Finite Precision RNNs for Language Recognition”) who exhibited basic limitations of GRU when compared to LSTM.
- Is the result by Weiss et al actually related to the result of the authors who found that 2 GRUS cannot accurately a line attractor without near zero constant curvature in the phase space?
Section 2
-Wrong brackets in equation (4)
-Missing bracket before citing Laurent & Brecht
Section 4
-”We conjecture that the system depicted in figure 2A..” Should be figure 3A
- Lemma1: UZ has capital Z subscript
Section 5.2
-”The added affine transformation allows for a sufficiently long subinterval”: “sufficiently long” is too vague
Section 5.3
“A manifold with without near zero constant curvature”: should be “a manifold without near zero constant curvature”
Appendix A
-Wrong brackets in equation (20)
Appendix B
- In the proof of Theorem 1, the derivative is of (29), not of (12)
Appendix C
Figure 9: “who’s initial conditions” should be “whose initial conditions”
After rebuttal:
It's better now. However, the revised introduction still says:  "GRU has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017) ... LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017).... specifically unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018)."
So better remove the first statement on Britz et al: "GRU has become wildly popular ... in machine translation (Britz et al., 2017)" because they actually show why GRU is NOT wildly popular in machine translation, as correctly justified later in the same paragraph.
Pending the above revision, we'd like to increase our evaluation by 2 points, up to 6!


There are many ways to translate a sentence into another language. Explicit modeling of such uncertainty may enable better model fitting to the data and it may enable users to express a preference for how to translate a piece of content. Latent variable models are a natural way to represent uncertainty. Prior work investigated the use of multivariate continuous and discrete latent variables, but their interpretation and use for generating a diverse set of hypotheses have been elusive. In this work, we drastically simplify the model, using just a single multinomial latent variable. The resulting mixture of experts model can be trained efficiently via hard-EM and can generate a diverse set of hypothesis by parallel greedy decoding. We perform extensive experiments on three WMT benchmark datasets that have multiple human references, and we show that our model provides a better trade-off between quality and diversity of generations compared to all baseline methods.

# Summary of model
The paper proposes a mixture model formulation of NMT where the mixing coefficients are uniform and fixed. The authors then proceed to derive a lowerbound on the marginal likelihood
p(y|x) = \sum_z p(z)p(y|x,z) > 1/K \max_z p(y|x,z)
by picking the component z for which the joint likelihood is maximised. With a uniform p(z) this clearly selects the z for which the conditional p(y|x,z) is maximum. I use strictly greater here because p(z) > 0 and p(y|x,z) > 0 for every z.
The loss L(\theta|x,y) for an observation (x,y) is \min_z - \log p(y|z,x; \theta)
whose gradient with respect to NN parameters (theta) is \grad_theta \log p(y|z,x; \theta) for the component z that minimises the negative log-conditional and 0 for every other component, thus while this requires K forward passes (to solve \min_z), it only takes 1 backwards pass.
# Discussion
I appreciate model-based (as opposed to search-based) attempts to improve diversity for generation tasks such as MT. Latent variable modelling aims at a more explicit account of the generative procedure, namely, the joint distribution, which can potentially disentangle and explain different modes of the marginal. Thus from that point of view, this paper points to an exciting direction. That said, in my view, the assumptions behind the proposed approach are not justifiable and some of the claims are simply not appropriate. Below I try to support this view.
A stepping stone of this model is that p(y|x,z) must be "large for only one value of z" (as authors put it), and authors *assume* that will be the case.
While the bound in equation (2) holds, whether or not p(y|z,x) turns out to be "large for only one value of z", it will be a very loose bound unless that happens.
The key point is that one cannot *assume* it to be the case. One could perhaps *promote* it to be the case, but there's no aspect of the model formulation (or objective) that promotes such behaviour.
Backpropagating through whichever component happens to assign the largest likelihood does not guarantee (nor encourages) the other conditionals to *independently* end up going to zero.
Given the level of parameter sharing, I'd even consider the possibility that the exact opposite happens. As authors put it themselves
"Instead, by sharing parameters, even unpopular experts receive some gradients throughout training."
It's true they do, but they are being updated on the basis of the unilateral opinion of the selected component about the likelihood of the data.
Note that the true posterior p(z|x,y) is exactly proportional to the likelihood, as the prior is *uniform and fixed*:
  p(z|x,y) \propto 1/K p(y|x,z) \propto p(y|x,z)
This means that the authors expect the likelihood to do component allocation on its own. That is, the conditionals p(y|x,z=1), ..., p(y|x,z=K) must somehow coordinate themselves in making good use of the latent components. Without any mechanism to promote "competition" (in the parlance of Jacobs et al 1991), I don't see how this can work.
Also, the paper claims to model uncertainty, if I take the posterior to fulfil this claim, then I'm just left with a likelihood (again, due to uniform prior). In any case, a notion of uncertainty here would be conditioned on a point estimate of the network's parameters and should thus be worded carefully.
# Clarifications
1. "we aim to explicitly model uncertainty during training" can you make a case for where that happens in your model?
2. "prevents the gating from training well and the latent variable embeddings from specializing" which gating?
3. "While they showed improvements due to the regularization effect of the Monte Carlo gradient estimate”. I find it strange to talk about the “regularisation effect” of a gradient estimate, perhaps you can be a bit more precise here? Or perhaps you are referring to some specific component of the objective function whose gradient we are estimating via MC and perhaps that component may have some regularisation effect.
4. if you aim to have p(y|x,z) high for a single latent variable at a time, you are implicitly saying that every x has at most (or rather exactly) K translations with non-negligible probability. Is that sensible?
# Pros/Cons
Pros
* simple: the approach presented here requires no significant changes to otherwise standard architectures, it instead concentrates in a change of objective and training algorithm.
* assessment of variability in translation: this paper proposes to use BLEU and a corpus of multiple references in an interesting (potentially novel) way.
Cons
* problematic assumptions: e.g. posterior will turn out sparse without any explicit way to promote such behaviour
* unrealistic claims: e.g. modelling uncertainty
* imprecise use of technical language: some technical terms are not used in their strictly technical sense (e.g. uncertainty, degeneracy), some explanations employ loosely defined jargons (e.g. regularisation effect of the gradient estimate)


MULTILINGUAL NEURAL MACHINE TRANSLATION WITH KNOWLEDGE DISTILLATION
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Summary: Train a multilingual NMT system using the technique of Johnson et al (2017), but augment the standard cross-entropy loss with a distillation component based on individual (single-language-pair) teacher models. Periodically compare the validation BLEU score of the multilingual model with that of each individual model, and turn off distillation for language pairs where the multilingual model is better. On three different corpora (IWSLT, WMT, TED) with into-English translation from numbers of source languages ranging from 6 (WMT) to 44 (TED), this technique outperforms standard distillation for every language pair, and outperforms the individual models for most language pairs. Supplementary experiments justify the strategy of selectively turning off distillation, and quantify the effect using only the top 8 vocabulary items in distillation.
The main idea makes sense, and the results are very convincing, especially since it appears that hyper-parameters were not tuned extensively (eg, weight of 0.5 on the distillation loss, for all language pairs). Implementation should be very straightforward, especially with the trick of pre-computing top-k probabilities from the teacher model at each corpus position. One small barrier to practical application that the authors fail to acknowledge is the requirement to train individual models, which will at least double training time compared to a single multilingual model.
The main missing experiment is higher-capacity multilingual models, which Johnson et al show to be beneficial in settings with a large number of language pairs. Using a multilingual model of the same (relatively small) size as the individual models as is done here is likely to be suboptimal, especially for the 44-language pair TED setting. A related point is that the corpora used seem to be quite small (eg 4.5M and 1M sentences for WMT Czech and German, respectively, while the available training corpora are closer to 15M and 4.5M). Although performance relative to individual models is still impressive - and seems to be better than than in previous work - this makes the experiments comparing to the multilingual baseline less meaningful.
Also missing are experiments on out-of-English translation, which would establish the viability of the proposed technique for many-to-many translation via bridging. Out-of-English is a more difficult problem than into-English. I can’t see any reason the proposed technique wouldn’t also work in this setting, but this remains to be shown.
Although it’s great that the technique is shown to work without embellishments, there are a few obvious strategies it would have been interesting to explore, such as making the weight on the distillation loss dependent on the difference in performance between the multilingual and individual models; and allowing for the distillation loss to be turned back on if the performance of the multilingual model starts to drift back down for a particular language pair. I also wondered about the effect of the gradient accumulation strategy in algorithm 1, where individual batches from each language pair are effectively grouped into one giant batch for the purpose of parameter updates. I can see that this could stabilize training, but it would be good to know whether it’s crucial for success, especially when the number of language pairs is large.
Further details:
As aforementioned -> As mentioned
(1) 2nd line: Doesn't make sense as written. You need to distinguish the gold
y_t from hypothesized ones in the 1() function.
Above (2): is served as -> serves as
3.2 First paragraph. Since D presumably consists of D^l for all languages l,
L_ALL(D,...) should be a function of teacher parameters theta^l for all
languages l rather than just one as written.
In top-K distillation, is the teacher distribution renormalized or simply
truncated?
Generalization analysis, pg 8: presumably you are sampling from N(0, sigma^2) -
this should be described as such.
Reference:
Johnson et al, “Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation” TACL, 2017.


Bayesian Policy Optimization for Model Uncertainty
Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.

Evaluation:
This is a solid paper: The idea is clear, it is well communicated and put into context of the existing literature, and the results are promising. The experiments are well chosen and illustrate the method well. The connection between the chosen setting (BAMDPs) to POMDPs is explained well and explored in the empirical evaluation as well. I think that the methods section could go into a bit more detail, and the underlying assumptions that the authors make could be discussed more critically.
Summary:
This paper looks at Bayes-Adaptive MDPs (BAMDPs) in which the latent parameter space is either
- a discrete finite set or
- a bounded continuous set that can be approximated via discretization.
Consequently, the authors choose to represent the belief as a categorical distribution, which can be represented by a vector of weights.
They further assume that the environment model is known. Hence, the posterior belief can be computed exactly.
If I understand correctly, the main contribution is that the authors represent the policy as a neural network and train it using a policy gradient algorithm.
This is a good first step towards scalable Bayesian policy optimisation.
Main Feedback:
- In the Introduction, first paragraph, you say one of the aspects of real-world robotics is that there's "(1) an underlying dynamical system with unknown latent parameters". I would argue that the dynamic system itself is typically also unknown, including how it is parametrized by these latent parameters. I think it is important to point this out more explicitly in the introduction (it is mentioned in sec 2 and 5, but maybe it's worth mentioning it in 4 again as well): for the problems that you look at, you assume that the form of the transition function is known (just not its parameters phi).
- In the main methods section (4), it would be nice to see some more detail about the Bayes filter. Can you write out the distribution over the latent parameters, and write out how the filtering is done? Explain how to compute the normalising constant (and mention explicitly why this is possible for your set-up, and why it would be infeasible if the latent space cannot be discretized). How exactly is the posterior distribution represented and fed to the policy? Seeing this done explicitly in Section 4 (even if it repeats some things that are explained in 2) would help someone that is interested in (re-)implementing the proposed method.
- I would like to see a more critical discussion in Section 7 about the assumptions that the authors make: that the environment models are known, and that the latent space can be discretized. How realistic are those assumptions (and in which kind of real-world problems can we make them), and what are ways forward to drop these assumptions?
Other Comments:
- Introduction: Using an encoder for the state/belief is an implementation choice, and (as I see it) not part of the main contribution. I would focus on explaining the intuition behind BPO in the introduction, and only mention the architecture choice as a side note.
- Related Work: The authors might be interested in the recent work of Igl et al. (ICML 2018, "Deep Variational RL for POMDPs"), who approximate the belief in a POMDP using variational inference and a particle filter.
Significance for ICLR:
- In the light-dark experiment, the authors visualise the belief that the agent has at every time step. It would have been nice to see an analysis of how exactly the belief looks also for maybe 1-2 other experiments, and how (when) the agent makes a decision based on this. This could replace Table 2 (which I guess should be called Figure 2?), which I did not find very insightful.