LEARNING SPARSE RELATIONAL TRANSITION MODELS
We present a representation for describing transition models in complex uncertain domains using relational rules. For any action, a rule selects a set of relevant objects and computes a distribution over properties of just those objects in the resulting state given their properties in the previous state. An iterative greedy algorithm is used to construct a set of deictic references that determine which objects are relevant in any given state. Feed-forward neural networks are used to learn the transition distribution on the relevant objects' properties. This strategy is demonstrated to be both more versatile and more sample efficient than learning a monolithic transition model in a simulated domain in which a robot pushes stacks of objects on a cluttered table.

SUMMARY:
This work is about learning state-transition models in complex domains represented as sets of objects, their properties, ``"deictic" reference functions between sets of objects, and possible actions (or action templates). A parametric model for the actions is assumed, and these parameters act on a neural net that learns the transition model (probabilistic rule) from the current state to the next one.  It is basically this nonlinear transition model implemented by a network which makes this work different from previous models described in the literature. The relational transition model proposed is sparse, based on the assumption that actions have only ``local effects on related objects. The prediction model itself is basically a Gaussian distribution whose mean and variances are represented by neural nets. For jointly learning multiple rules, a clustering strategy is presented which assigns experience samples to transition rules. The method is applied to simulated data in the context of predicting pushing stacks of blocks on a cluttered table top.
EVALUATION:
The type of problems addressed in this paper is challenging and highly relevant for solving problems in the ``real'' world. Although the method proposed is in some sense a direct generalization of the work in [Pasula et al.], it still contains many novel and interesting aspects.Any single part of the model (like the use of Gaussians parametrized by functions implemented via neural nets) is somehow ``standard in deep latent variable models, but in complex real-world rule-learning problems the whole system presented  defines  certainly a big improvement over the state-of-the-art, which in my opinion has the potential to indeed advance this field of research.


Recent deep multi-task learning (MTL) has been witnessed its success in alleviating data scarcity of some task by utilizing domain-specific knowledge from related tasks. Nonetheless, several major issues of deep MTL, including the effectiveness of sharing mechanisms, the efficiency of model complexity and the flexibility of network architectures, still remain largely unaddressed. To this end, we propose a novel generalized latent-subspace based knowledge sharing mechanism for linking task-specific models, namely tensor ring multi-task learning (TRMTL). TRMTL has a highly compact representation, and it is very effective in transferring task-invariant knowledge while being super flexible in learning task-specific features, successfully mitigating the dilemma of both negative-transfer in lower layers and under-transfer in higher layers. Under our TRMTL, it is feasible for each task to have heterogenous input data dimensionality or distinct feature sizes at different hidden layers. Experiments on a variety of datasets demonstrate our model is capable of significantly improving each single task's performance, particularly favorable in scenarios where some of the tasks have insufficient data.

Summary:  This paper studies deep multi-task learning. Prior papers have studied various knowledge sharing approaches for deep multi-task learning including hard and soft sharing schemes. And some soft sharing schemes have used tensor decompositions (including TT, and Tucker). This paper fuses this line of work with the recently proposed Tensor-Ring decomposition in order to obtain Tensor Ring (TR)-based soft sharing for multi-task learning. The results show some improvement over prior deep MTL methods based on other tensor factorisation methods.
Strengths:
+ Nice extension of existing line of work tensor-factorisation based MTL.
+ More flexibility for controlling shared/unshared portions of weights compared to DMTRL.
+ Improves on previous methods results.
+ Experiments evaluate how MTL methods relate with various amounts of training data on each task.
Weaknesses:
- Novelty/significance is limited.
- Writing. Many things are not clearly and intuitively explained. Some claims are not adequately justified.
- Introduces more hyper parameters to tune.
- Results may rely on hyper parameter tuning.
Comments:
1. Novelty.  Existing studies already established the template of different tensor factorisation methods (TT, Tucker) being possible to plug into deep networks for different kinds of soft-sharing MTL. Meanwhile, TR decomposition is taken off the shelf; (and as it’s been applied for compression before, this is not the first time TR decomposition has been used in a CNN context either). Therefore this is an A+B paper and a high bar should be met for the additional analysis, insight, or performance improvements that should provided.
2. Lots of writing issues:
2.1 Many things are not explained transparently enough at best (or major over-claim at worst). For example:
2.1.1 Paper claims the benefit that each task can have its own I/O dimensionality. However if TR-decomp is “circularly connected” TT-decomp (Fig 1), then this seems not to happen automatically. So it should be unpacked more clearly how this is achieved.
2.1.2 Paper claims favourable ability to use more private cores than TT, where only one core is private. However circular TT would also seem to have one private core by default (the core with a task axis). So I suspect something else is going on, but this is completely unclear and should be explained more transparently. Furthermore it should be justified if whatever modifications do enable these properties are definitely a unique property of TR-decomp, or could also be applied to TT-decomp.
2.1.3 Statement “TRMTL generalizes to allow the layer-wise weight to be represented by a relatively lager number of latent cores” unclear: generalises what? larger number of cores than what? Than TT? The previous presentation suggests TT and TR should have same number of cores.
2.1.4 Statements like “TR enjoys the property of circular dimensional permutation invariance” are made without any explanation about what is the implication of this for neural networks and multi-task learning.
2.2 Many claims are inaccurate or not adequately backed up by theory or experiment. EG: (i) Paper claims to include DMTRL as a special case. But it only subsumes DMTRL-TT, not DMTRL-Tucker. Because TR-decomp does not include Tucker-decomp as an exact special case.  (ii) Sentences “TR-ranks are usually smaller than TT-ranks” are assertions without verification.
2.3 Sentences are taken verbatim from other papers, plagiarism. For example: “TR model is more flexible than TT, because TR-ranks can be equally distributed in the cores, but TT-ranks have a relatively fixed pattern”  is verbatim from Zhao’16 TR-decomp paper.
3. Hyperparameters: This paper apparently gains some practical benefit due to the notion of shared/unshared cores. However, this also introduces  additional hyper parameters (E.g., each layers private proportion “c”) to tune besides the ranks. Unlike the rank that can be pre-estimated by reconstruction error, this one seems to require tuning by cross-validation. This is not scalable.
4. Hyperparameters+Tuning: Hyperparameters Private proportion, “sharing pattern”, IO dimension seem to be tuned by accuracy.( “We test different sharing patterns and report the ones with the best accuracies”). This is even less scalable, and additional tuning makes it unsurprising it surpasses other models performance.
5. Insight & Analysis. All the core selection & public/private core selection are treated as black box optimisation. No insight is given about what turns out to be useful to share or not, and how consistent this is, etc.


3D objects (artefacts) are made to fulfill functions. Designing an object often starts with defining a list of functionalities that it should provide, also known as functional requirements. Today, the design of 3D object models is still a slow and largely artisanal activity, with few Computer-Aided Design (CAD) tools existing to aid the exploration of the design solution space. The purpose of the study is to explore the possibility of shape generation conditioned on desired functionalities. To accelerate the design process, we introduce an algorithm for generating object shapes with desired functionalities. We follow the principle form follows function, and assume that the form of a structure is correlated to its function. First, we use an artificial neural network to learn a function-to-form mapping by analysing a dataset of objects labeled with their functionalities. Then, we combine forms providing one or more desired functions, generating an object shape that is expected to provide all of them. Finally, we verify in simulation whether the generated object possesses the desired functionalities, by defining and executing functionality tests on it.

This paper is addressing several research challenges as a method to generate objects with desired functionalities, a method to extract form-to-function mapping, a method to operationally support a functionality arithmetic. The work illustrated in this paper is really interesting and is addressing relevant and open problems in the domain of product design.
Nevertheless the manuscript has a couple of weaknesses, one concerned with the presentation and another related to the design of the study.
The lack of a consistent choice for the lexicon is sometimes misleading. It is not always clear whether the use of different terms is addressing synonyms or to discriminate between two distinct concepts. For example let consider the following pairs: functionality versus affordance, function versus functional, class versus category, feature versus shape.
The study addresses several questions. Not always is clear what is the purpose or better the research questions that are driving the design of the experiments. While in the manuscript the are many repetition of the objectives of the study, less attention is devoted to explain what are the working hypothesis underlying the proposed methods. For example, one of the objective is a method to generate objects with desired functionalities. Only in the final Section there is a brief mention of the dichotomy between meash-based versus voxel-based. As reported in Section 2 there are in literature other works but there is not a claim on what is the specific purpose of the present study. The contrast of voxel versus mesh looks like a motivation but it only a speculation. A similar comment might address the dichotomy deterministic (ontology) versus probabilistic (autoencoder). In this case the experiment design should provide some empirical evidence about this contrast.
A minor comment. Figure 7a is illustrating the functional essence of table. According to the caption Figure 5a is illustrating the same functional essence for the same category/class table. Should the pictures look the same?


WOULDA, COULDA, SHOULDA: COUNTERFACTUALLY-GUIDED POLICY SEARCH
Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for modelbased policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, i.e. actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.

Summary: by assuming a correct, strongly factored environment model, improved estimators useful for policy search can be derived by "counterfactual reasoning", where data sampled from experience is used to refine initial conditions in the model; this translates into improved estimators of policy values, which improves policy search.
Major comments:
I enjoyed this paper.  I think that model-based RL deserves more work, and I think that this is a simple, reasonably workable approach with some nice theoretical benefits.  I like the idea of SCMs; I like the idea of counterfactual reasoning; I like the idea of leveraging models in this unique way.
On the negative side, I felt that the paper makes some rather strong assumptions - specifically, that the agent has access to a perfect model with no mismatch, and that the model decomposes neatly into noise variables plus deterministic functions.  Given such a model, one wonders if there are other techniques, say, from classical planning, that could also be used for some sort of policy search.
I have a few questions about approximations.  First, I see that probabilistic inference is a core element of each algorithm (where p(u|h) must be computed).  For large, complex models, I assume this must be approximate inference.  This leads naturally to questions about accuracy (does approximate inference result in biased estimators? [probably yes]), efficacy (do the inaccuracies inherent in approximate inference outweigh the benefits of using p(u|h) vs. p(u)?) and scalability (how large of a model can we reasonably cope with before degradation is unacceptable, or no better than non-CF algorithms?).  As far as I can tell, none of this was addressed in the paper, although I do not expect every paper to answer every question; this is a first step.
I wish the experiments were a little more varied.  The experimental results really only show marginal improvement in one small task.  While I understand that this is not an empirical paper, neither does it fit strongly into the category of "theory paper".  For example, there are no theory results indicating what sort of benefit we might expect from using the methods outlined here, and in the absence of such theory, we might reasonably look to various experiments to demonstrate its effectiveness.
Pros:
+ Integration with SCMs is interesting
+ Counterfactual variants of algorithms are clearly motivated and interesting
+ Paper is generally well-written
Cons:
- Assumption that the agent is given a model with no mismatch is very strong
- Model class (noise variables + deterministic functions) seems potentially restrictive
- Questions about impact of approximate inference
- Experiments could have been more varied


LEARNING CONCISE REPRESENTATIONS FOR REGRES- SION BY EVOLVING NETWORKS OF TREES
We propose and study a method for learning interpretable representations for the task of regression. Features are represented as networks of multi-type expression trees comprised of activation functions common in neural networks in addition to other elementary functions. Differentiable features are trained via gradient descent, and the performance of features in a linear model is used to weight the rate of change among subcomponents of each representation. The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation. We compare several stochastic optimization approaches within this framework. We benchmark these variants on 100 open-source regression problems in comparison to state-of-the-art machine learning approaches. Our main finding is that this approach produces the highest average test scores across problems while producing representations that are orders of magnitude smaller than the next best performing method (gradient boosting). We also report a negative result in which attempts to directly optimize the disentanglement of the representation result in more highly correlated features.

The paper proposes a method for learning regression models through evolutionary
algorithms that promise to be more interpretable than other models while
achieving similar or higher performance. The authors evaluate their approach on
99 datasets from OpenML, demonstrating very promising performance.
The authors take a very interesting approach to modeling regression problems by
constructing complex algebraic expressions from simple building blocks with
genetic programming. In particular, they aim to keep the constructed expression
as small as possible to be able to interpret it easier. The evaluation is
thorough and convincing, demonstrating very good results.
The presented results show that the new method beats the performance of existing
methods; however, as only very limited hyperparameter tuning for the other
methods was performed, it is unclear to what extent this will hold true in
general. As the main focus of the paper is on the increased interpretability of
the learned models, this is only a minor flaw though.
The interpretability of the final models is measured in terms of their size.
While this is a reasonable proxy that is easy to measure, the question remains
to what extent the models are really interpretable by humans. This is definitely
something that should be explored in future work, as a small-size model does not
necessarily imply that humans can understand it easily, especially as the
generated algebraic expressions can be complex even for small trees.
The description of the proposed method could be improved; in particular it was
unclear to this reviewer why the features needed to be differentiable and what
the benefit of this was (i.e. why was this the most appropriate way of adjusting
weights).
In summary, the paper should be accepted.


SUBGRADIENT DESCENT LEARNS ORTHOGONAL DIC- TIONARIES
This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem. We show that a subgradient descent algorithm, with random initialization, can recover orthogonal dictionaries on a natural nonsmooth, nonconvex 1 minimization formulation of the problem, under mild statistical assumption on the data. This is in contrast to previous provable methods that require either expensive computation or delicate initialization schemes. Our analysis develops several tools for characterizing landscapes of nonsmooth functions, which might be of independent interest for provable training of deep networks with nonsmooth activations (e.g., ReLU), among other applications. Preliminary synthetic and real experiments corroborate our analysis and show that our algorithm works well empirically in recovering orthogonal dictionaries.

This paper is a direct follow-up on the Sun-Qu-Wright non-convex optimization view on the Spielman-Wang-Wright complete dictionary learning approach. In the latter paper the idea is to simply realize that with Y=AX, X being nxm sparse and A a nxn rotation, one has the property that for m large enough, the rows of X will be the sparsest element of the subspace in R^m generated by the rows of Y. This leads to a natural non-convex optimization problem, whose local optimum are hopefully the rows of X. This was proved in SWW for *very* sparse X, and then later improved in SQW to the linear sparsity scenario. The present paper refines this approach, and obtain slightly better sample complexity by studying the most natural non-convex problem (ell_1 regularization on the sphere).
I am not an expert on SQW so it is hard to evaluate how difficult it was to extend their approach to the non-smooth case (which seems to be the main issue with ell_1 regularization compared to the surrogate loss of SQW).
Overall I think this is a solid theoretical contribution, at least from the point of view of non-smooth non-convex optimization. I have some concerns about the model itself. Indeed *complete* dictionary learning seemed like an important first step in 2012 towards more general and realistic scenario. It is unclear to this reviewer whether the insights gained for this complete scenario are actually useful more generally.


We consider learning based methods for visual localization that do not require the construction of explicit maps in the form of point clouds or voxels. The goal is to learn an implicit representation of the environment at a higher, more abstract level, for instance that of objects. We propose to use a generative approach based on Generative Query Networks (GQNs, Eslami et al., 2018), asking the following questions: 1) Can GQN capture more complex scenes than those it was originally demonstrated on? 2) Can GQN be used for localization in those scenes? To study this approach we consider procedurally generated Minecraft worlds, for which we can generate images of complex 3D scenes along with camera pose coordinates. We first show that GQNs, enhanced with a novel attention mechanism can capture the structure of 3D scenes in Minecraft, as evidenced by their samples. We then apply the models to the localization problem, comparing the results to a discriminative baseline, and comparing the ways each approach captures the task uncertainty.

**Summary of the paper**
This paper studies the problem of visual re-localization, where we are interested in estimating the camera pose of a new image from a set of source images and their camera poses. Instead of explicitly designing the structure of a map of 3D scenes (e.g. occupancy grids or point clouds), the paper proposes implicitly learning an abstract map representation. Specifically, the paper proposes a generative method based on Generative Query Networks (GQNs) augmented with an attention mechanism. The authors apply this model to the visual re-localization problem. To train and test the proposed model, the authors introduce the Minecraft random walk dataset, which consists of images and their camera poses extracted from randomly generated trajectories in the Minecraft environment. The proposed model is compared against a discriminative counterpart, which is trained to directly predict the target camera pose and achieves better MSE.
**Clarity**
Above average
**Significance**
Below Average
**Detailed comments**
_Paper Strengths_
- The idea of leveraging generative models' knowledge of "maps" to perform visual localization is interesting. This gives learning frameworks the flexibility of building a latent representaiton of maps which may yield better performance instead of being restricted to a pre-defined representations.
- The paper is very well-written and easy to follow.
- The authors did a good job presenting the proposed methods. The descriptions and formulations are clear. Both Figure 2 and Figure 3 are helpful for understanding the GQNs and the proposed attention mechanism.
- The patch dictionary for the attention mechanism seems effective especially when dealing with a set of context images capturing the same scene.
- The authors are honest about the limitations of the proposed framework compared to classic approaches.
- The visualization of results are clear. Particularly, Figure 5 and Figure 7 give easily interpretable representations of the results.
_Paper Weaknesses_
- Implicitly learning a map of the scene is mentioned as a strength in the paper, but this comes at the high cost of interpretability. Without an explicit map representation, it is difficult to understand the failure cases - does the model not understand the 3D scene well or does the model have a hard time accurately predicting camera poses?
- Minecraft is an interesting environment for proof of concept, but lacks much of the subtlety of the real world.
- Building a framework that is able to perform the localization task from real-world scenes is more interesting. Learning generative models of real-world scenes is known to be difficult, which makes this framework impractical. There are google streetview and indoor datasets authors can try to utilize.
- The aforementioned point is supported by the fact that the localization performance of the proposed model on real-world scenes is missing.
- The reviewer does not find enough novelty from the proposed model, which is an iterative improvement on GQNs.
- The paper only compares the proposed model against its discriminative couterpart, which is not sufficient. While the authors strongly argue that exploiting the proposed implicitly representations of scenes is more beneficial than utilizing the pre-defined explicit representations, the only baseline is using the same implicit representations. Although the reviewer is aware of that this model does not use complete video sequences, benchmarking against a visual monocular SLAM algorithm, like LSD-SLAM [1], would contextualize the claim.
- Why quantize the discriminative model's output? This de-correlates nearby pose values. The paper could benefit from an explanation of not using a straightforward regression over pose variables.
- Overall, the reviewer does not find enough novelty from any aspects except the idea of utilizing a generative model for visual localization with implocitly learned maps, which is not fully demonstrated in the experiment section (i.e. not compare to baselines using explicit maps).
- A differentiating factor for this paper could be tackling one of the open problems remaining in SLAM as identified in [2], like lifelong learning or semantic mapping.
_Reproducibility_
- Given the clear description in the main paper and the details provided in the appendix, the reviewer believes reproducing the results is possible.
_Conclusion_
- Overall, the reviewer believes this paper is well presented and reproducible. However, the paper does not propose to solve a novel problem, nor does it present a very novel method. Although the idea of using existing generative networks for localization is interesting, the paper misses important baselines relying on explicit map representation and is not sufficiently convincing. Moreover, requiring a generative model significantly limits the possibility of utilizing the proposed model for real-world applications. While the paper does present a new dataset built in Minecraft which is suitable for demonstrating the strengths of the proposed method, the reviewer does not find this significant. Therefore, the reviewer recommends a rejection.
_Reference_
[1] Engel, Jakob, Thomas Schöps, and Daniel Cremers. "LSD-SLAM: Large-scale direct monocular SLAM." European Conference on Computer Vision. Springer, Cham, 2014.
[2] Cadena, Cesar, et al. "Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age." IEEE Transactions on Robotics 32.6 (2016): 1309-1332.


ACCUMULATION BIT-WIDTH SCALING FOR ULTRA- LOW PRECISION TRAINING OF DEEP NETWORKS
Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices. This imposes an upper-bound on the reduction of complexity of multiply-accumulate units. We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training. Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation. We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet. In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline. We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds. Overall this analysis enables precise tailoring of computation hardware to the application, yielding area-and power-optimal systems.

There has been a lot of work on limited precision training and inference for deep learning hardware, but in most of this work, the accumulators for the multiply-and-add (FMA) operations that occur for inner products are chosen conservatively or treated as having unlimited precision. The authors address this with  an analytical method to predict the number of mantissa bits needed for partial summations during the forward, delta and gradient computation ops for convolutional and fully connected layers. They propose an information theoretic approach to argue that by using fewer bits of mantissa in the accumulator than necessary, the variance of the resulting sum is less than what it would have been if sufficient bits of mantissa were used. This is surprising to me, as quantization is usually modeled as _adding_ noise, leading to an _increase_ in variance (Mc Kinstry et al. 2018), so this is a nice counterexample to that intuition. Unfortunately the result is presented in a way that implies the variance reduction is what causes the degradation in performance, while obviously (?) it's just a symptom of a deeper problem. E.g., adding noise or multiplying by a constant to get the variance to where it should be, will not help the network converge. The variance is just a proxy for lost information. The authors should make this more clear.
Loss of variance is regarded as a proxy to the error induced/loss of information due to reduced mantissa prevision. The authors present their metric called Variance Retention Ratio (VRR) as a function of the mantissa length of product terms, partial sum (accumulator) terms, and the length of the accumulation. Thereafter, the mantissa precision of the accumulator is predicted to maintain the error of accumulation within bounds by keeping the VRR as close to 1 as possible. The authors use their derived formula for VRR to predict the minimum mantissa precision needed for accumulators for three well known networks: AlexNet, ResNet 32 and ResNet 18. For tightness analysis they present convergence results while perturbing the mantissa bits to less than those predicted by their formula, and show that it leads to more than 0.5% loss in the final test error of the network.
Some questions that the manuscript leaves open in it's current form:
0. Does this analysis only apply to ReLu networks where all the accumulated terms are positive? Would a tanh nonlinearity, e.g. in an RNN, result in a different kind of swamping behavior? I don't expect the authors to add a full analysis for the RNN case if it's indeed different, but it would be nice to comment on it.
1. Do the authors assume that the gradients and deltas will always be within the exponent range of representation? I do not find a mention of this in the paper. In other words, are techniques like loss scaling, etc. needed in addition? Other studies in literature analyzing IEEE fp16 seem to suggest so.
2. The authors do not provide details on how they actually performed the experiments when running convergence experiments. It is not straightforward to change the bit width of the accumulator mantissa in CPU or GPU kernel libraries such as CUDNN or Intel MKL. So how do they model this?
3. On page 7, the authors point out that they provide a theoretical justification of why the chunk size should neither be too small or too large - but I do not see such a justification in the paper. More detailed explanation is needed.
There are a few minor typos at a few places, e.g.

1. Page 4: “… , there is a an accumulation length….”
2. Page 6: “…floaintg-point format…"
Some figures, notably 2 and 5, use text that is unreadably small in the captions. I know this is becoming somewhat common practice in conference submissions with strict pages limits, but I implore the authors to consider shaving off space somewhere else. Some of us still read on paper, or don't have the best eyes!


ADVERSARIAL REPROGRAMMING OF NEURAL NETWORKS
Deep neural networks are susceptible to adversarial attacks. In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer. Previous adversarial attacks have been designed to degrade performance of models or cause machine learning models to produce specific outputs chosen ahead of time by the attacker. We introduce attacks that instead reprogram the target model to perform a task chosen by the attacker-without the attacker needing to specify or compute the desired output for each test-time input. This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary-even if the model was not trained to do this task. These perturbations can thus be considered a program for the new task. We demonstrate adversarial reprogramming on six ImageNet classification models, repurposing these models to perform a counting task, as well as classification tasks: classification of MNIST and CIFAR-10 examples presented as inputs to the ImageNet model.

This paper proposed "adversarial reprogramming" of well-trained and fixed neural networks, which can be viewed as learning a trainable input perturbation on a fixed network for multi-tasking by using a different dataset (e.g., MNIST) from the original dataset (ImageNet) as input. Domain mapping functions (h_g and h_f) are required if the data have different dimensions. The key factor to enable adversarial reprogramming of a fixed network to perform a different task is by training the additive adversarial program as defined in (1). Experimental results show that 7 different ImageNet models (adversarially trained or not) can be reprogrammed for performing counting tasks, and MNIST and CIFAR-10 classifications. The authors also show that adversarial reprogramming is less effective on untrained networks.
Although the idea of this paper is interesting,  the contribution is unclear and the "adversarial" setting is not well motivated. The detailed comments are as follows.
1. Unclear contribution - As mentioned in this paper, the main difference between "adversarial reprogramming" and transfer learning or multi-task learning is the fact that the network to be reprogrammed is fixed during reprogramming and was trained on a single task that is independent of the targeted task. However, the reprogramming results are not surprising given the fact that multi-task learning can be achieved on the same network. Given the fact that the perturbed input data (e.g., MNIST) is different from the original input data (ImageNet), what adversarial reprogramming demonstrates is actually a simple way of learning a new task via input perturbation to an unseen dataset at training time. However, transfer learning can be done in a similar way by simply fine-tuning the last (few) layers of a well-trained network. So the number of parameters required to be modified in order to "reprogram" a network is already known to be quite small via fine-tuning, which may even be less than the dimension of the adversarial program. In addition, given that the input of ImageNet model is high-dimensional and ImageNet images are likely to lie on a low dimensional manifold (but they are very different from hand-written digits or CIFAR images), the capability of reprogramming using deep models under this setting is expected and thus the contribution is unclear.
2. The "adversarial" setting is vague - I am very confused about why the experimental settings should be considered "adversarial", given the fact that ImageNet images and the three sets of adversarially perturbed images are quite different. What the experiments show is that a well-trained classifier has a large enough capacity to perform other tasks by simply training a perturbation on a different (out-of-distribution) dataset as inputs. It would make more sense to call this method "adversarial" if it can be used on ImageNet images to secretly implement some programmed tasks, while on the surface they are seemingly simply performing a typical classification task.
3. Limited novelty - How is adversarial program different from additional perturbation? Let alone the mapping function M in eqn (3), the adversarial program is nothing but a constrained perturbation (ranging from [-1,1] in each dimension). The optimization formulation in (3) can be seen as a  Carlini-Wager L2 attack with a simplified attack loss + L2 distortion regularization. Therefore, the proposed method has limited technical contribution and novelty.
In summary, this paper has some interesting ideas, but the current presentation lacks clear motivation, and its technical contribution and implications need to be better highlighted.  The authors are suggested to better motivate this paper from the angle of studying the learning capacity of input perturbation induced multi-tasking learning of a well-trained and fixed neural network model, and compare the pros and cons with transfer learning based on fine-tuning and joint multi-task learning / meta-learning on the same network architecture. Based on my own reading, I truly feel that advocating  "adversarial" reprogramming does not add any value to this work, as its use for an adversary is not properly motivated (e.g., visual imperceptibility) and its training has no adversarial nature (e.g., GAN training). Titles like "(Out-of-domain) Input perturbation induced reprogramming of neural networks" should better justify the contents and experiments presented in this work. Lastly, the authors need to specify how equation (3) is different from the formulation of finding adversarial perturbations in existing literature. Otherwise,  the novelty of "adversarial program" is quite limited.
----
Post-rebuttal review:
I appreciate the authors' efforts in including the new experiments in Sections 4.4 and 4.5. In my opinion, these new results and the discussion in Section 5.2 add great values to this work and make the contributions of this paper substantially clear. I've increased my rating to 6.


The ability to predict matches between two sources of text has a number of applications including natural language inference (NLI) and question answering (QA). While flexible neural models have become effective tools in solving these tasks, they are rarely transparent in terms of the mechanism that mediates the prediction. In this paper, we propose a self-explaining architecture where the model is forced to highlight, in a dependent manner, how spans of one side of the input match corresponding segments of the other side in order to arrive at the overall decision. The text spans are regularized to be coherent and concise, and their correspondence is captured explicitly. The text spans -rationales -are learned entirely as latent mechanisms, guided only by the distal supervision from the end-to-end task. We evaluate our model on both NLI and QA using three publicly available datasets. Experimental results demonstrate quantitatively and qualitatively that our method delivers interpretable justification of the prediction without sacrificing state-of-the-art performance. Our code and data split will be publicly available.

This paper tackles the problem of generating rationales for text matching problems (i.e., two pieces of text are given). The approach is in a similar spirit as (Lei et al, 2016) while the latter mainly focuses on one piece of text for text classification problems and this work focuses on generating pairs of rationales. The approach has been evaluated on NLI and QA datasets, which demonstrates that the generated rationales are sensible and comes at a cost of accuracy.
The approach employs a generation-encoding-generation schema: it firsts generates the rationale from one side as a sequence tagging problem, re-encodes the rationales and predicts the rationale on the other side as a span prediction problem. Leveraging a match-LSTM framework and generated rationales for prediction, the model can be trained using a policy gradient method.
Overall, I think this problem is novel and interesting. However, I am not fully convinced whether the proposed solution (and its implementation) is the right way to do so. Also, the paper writing needs to much improved.
First of all, there is certainly a drop in the end task performance while it is unclear whether the derived rationales are really that useful (if the goal is interpretability) in the current evaluation. I am not convinced by the noisy SciTrail evaluation for rationales -- the noisy part p’ can be totally irrelevant and assume that the rationale generation component learns some sort of alignment between two parts, so it is not surprising that the model will not select words from p’ and it doesn’t really show that the rationales are useful. I think it is necessary to conduct some human evaluation for generate rationales and also provide some simple baselines for comparison (for example, just converting the soft-attention in math-LSTM to some hard selections) and see if this interpretability (at a cost of task performance) is really worthy or not.
Secondly, I am not sure that whether the current way of generating the rationale pairs really makes sense or not.
It casts the rationale generation on one side as a tagging problem while the rationale generation on the other side as a span prediction problem. Why is that? Do you make any assumption that the two pieces of texts are not symmetric (e.g., one side is much longer than the other side like most of the current QA setup)?
There is a regularization term for both x and y but it seems that there isn’t any constraint that the generated rationales on the y side are not overlapping. Is it a problem or not? I don’t know how this is dealt with in the implementation.
Understanding sec 3 takes some efforts and I think the presentation could be much improved. For example, q * {x^k} is not defined -- I assume it means extracting the subset of q based on the 1’s in {x^k}. The equations in Sec 3.2 can be made clearer.
Finally, it is also unclear that how the 3 datasets were chosen. There are so many NLI and QA datasets (some of them are more popular and more competitive) at this point. Is there a reason that these datasets were chosen? There is a setup called ‘no rationalization w/ re-encoding’ which means that the rationale is already provided on one side, but is unclear that whether the OpenIE tuple and the searchQA queries can be used as rationales directly.
Minor points:
- Distal supervision -> distant supervision
- The first paragraph of Introduction, “absent attention or rationale mechanisms”, what does it mean by ‘absent attention’? Isn’t it the case that all the models used attention mechanisms?


BIOLOGICALLY-PLAUSIBLE LEARNING ALGORITHMS CAN SCALE TO LARGE DATASETS
The backpropagation (BP) algorithm is often thought to be biologically implausible in the brain. One of the main reasons is that BP requires symmetric weight matrices in the feedforward and feedback pathways. To address this "weight transport problem" (Grossberg, 1987), two biologically-plausible algorithms, proposed by Liao et al. (2016b) and Lillicrap et al. (2016), relax BP's weight symmetry requirements and demonstrate comparable learning capabilities to that of BP on small datasets. However, a recent study by Bartunov et al. (2018) finds that although feedback alignment (FA) and some variants of target-propagation (TP) perform well on MNIST and CIFAR, they perform significantly worse than BP on ImageNet. Here, we additionally evaluate the sign-symmetry (SS) algorithm (Liao et al., 2016b), which differs from both BP and FA in that the feedback and feedforward weights do not share magnitudes but share signs. We examined the performance of sign-symmetry and feedback alignment on ImageNet and MS COCO datasets using different network architectures (ResNet-18 and AlexNet for ImageNet; RetinaNet for MS COCO). Surprisingly, networks trained with signsymmetry can attain classification performance approaching that of BP-trained networks. These results complement the study by Bartunov et al. (2018) and establish a new benchmark for future biologically-plausible learning algorithms on more difficult datasets and more complex architectures.

This work adds to a growing literature on biologically plausible (BP) learning algorithms. Building off a study by Bartunov et al. that shows the deficiencies of some BP algorithms when scaled to difficult datasets, the authors evaluate a different algorithm, sign-symmetry, and conclude that there are indeed situations in which BP algorithms can scale. This seemingly runs counter to the conclusions of Bartunov et al.; while the authors state that their results are "complementary", they also state that the findings “directly conflict” with the results of Bartunov, concluding that BP algorithms remain viable options for both learning in artificial networks and the brain.
To reach these conclusions the authors report results on a number of experiments. First, they show successful training of a ResNet-18 architecture on ImageNet using sign-symmetry, with their model performing nearly as well as one trained with backpropagation. Next, they demonstrate decent performance on MS COCO object detection using RetinaNet. Finally, they end with a discussion that seeks to explain the differences in their approach and the approach of Batunov et al, and with a potential biological implementation of sign symmetry.
Overall the clarity of the writing is sufficient. The algorithm is properly explained, and there are sufficient citations to reference prior work. The results are generally clear (though there is an incomplete experiment, I agree with the authors that it is unlikely for the preliminary results to change). I believe that there is enough detail for this work to be reproducible. The work is also sufficiently novel in that experiments using sign-symmetry on difficult datasets have not been undertaken, to my knowledge.
Unfortunately, the clarity and rigor of the *scientific argument* is insufficient for a number of reasons. These will be enumerated below.
First, the explicit writing and underlying tone of the paper reveal a misrepresentation of the scientific argument in Bartunov et al. The scientific question in Bartunov et al. is not a matter of whether BP algorithms can be useful in purely artificial settings, but rather whether they can say anything about the way in which the brain learns. In this work, on the other hand, there seems to be two scientific questions: first, to assess whether BP algorithms can be useful in artificial settings, and second, to determine whether they can say anything about how the brain learns, as in Bartunov (indeed, the author’s conclusions highlight precisely these two points). Unfortunately, the experiments and underlying experimental logic push towards addressing the first question, and use this as evidence towards a conclusion to the second question. More concretely, experiments are run on biologically problematic architectures such as ResNet-18, often with backpropagation in the final layer (though admittedly this doesn’t seem to be an important detail with sign-symmetry, for reasons explained below). This is fine under the pretense of answering the first question, but to seriously engage with the results of Bartunov et al. and assess sign-symmetry’s merit as a BP algorithm for learning in the brain, the work requires the authors the algorithms to be tested under similar conditions before claiming that there is a “direct conflict”. To this end, though the authors claim that the conditions on which Bartunov et al tested are “somewhat restrictive”, this logic can equally be flipped on its head: the conditions under which this paper tests sign-symmetry are not restrictive enough to productively move in the direction of assessing sign-symmetry’s usefulness as a description of learning in the brain, and so the conclusion that the algorithm remains a viable option for describing learning in the brain is not sufficiently supported. On the other hand, I think the conclusions regarding the first question -- whether sign-symmetry can be useful in artificial settings -- are fine given the experiments.
Second, the work does not sufficiently weigh the “degree” of implausibility of sign-symmetry compared to the other algorithms, and implicitly speaks of feedback alignment, target propagation, and sign-symmetry as equally realistic members of a class of BP algorithms. Of course, one doesn’t want to go down the road of declaring that “algorithm A is more plausible than algorithm B!”, but the nuances should at least be seriously discussed if the algorithms are to be properly compared. In backpropagation the feedback connections must be similar in sign and magnitude. Sign-symmetry eliminates the requirement that the connections be similar in magnitude. However, this factor is arguably the least important of the two (the direction of the gradient is more important than the magnitudes), and we are still left with feedback weights that somehow have to tie their sign to their feedforward counterparts, which is not an issue in target propagation or feedback alignment. The authors try to explain away this difficulty with an appeal to molecular biology, which leads into my third point.
Third, the appeal to molecular mechanisms to explain how sign-symmetry can arise is not rigorous. There is a plethora of molecular mechanisms at play in our cells; indeed, there are enough mechanisms to hand-craft *any* sort of circuit one likes. Thus, it is somewhat vacuous to conclude that a particular circuit can be “easily implemented” in the brain simply by appealing to a hand-crafted circuit. For this argument to hold one needs to appeal to biological data to demonstrate that such a circuit either a) exists already, b) most probably exists because of reasons X, Y, Z. Unfortunately there is no biological backing, rendering this argument a possibly fun thinking exercise, but not a serious scientific proposal. But perhaps most problematic, the argument leaves the problem of sign-switching in the feedforward network to “future work”. This is perhaps *the most* important problem at play here, and until it is answered, these arguments don’t have sufficient impact.
Altogether the scientific argument of this work needs tightening. The tone, the title, and the overall writing should be modified to better tackle the nuances underlying the arguments of biologically plausible learning algorithms. The claims and conclusions need to be more explicit, and the work needs to better seated in the context of both the previous literature, and the important questions at play for assessing biologically plausible learning algorithms.


DOUBLE VITERBI: WEIGHT ENCODING FOR HIGH COMPRESSION RATIO AND FAST ON-CHIP RECON- STRUCTION FOR DEEP NEURAL NETWORK
Weight pruning has been introduced as an efficient model compression technique. Even though pruning removes significant amount of weights in a network, memory requirement reduction was limited since conventional sparse matrix formats require significant amount of memory to store index-related information. Moreover, computations associated with such sparse matrix formats are slow because sequential sparse matrix decoding process does not utilize highly parallel computing systems efficiently. As an attempt to compress index information while keeping the decoding process parallelizable, Viterbi-based pruning was suggested. Decoding non-zero weights, however, is still sequential in Viterbi-based pruning. In this paper, we propose a new sparse matrix format in order to enable a highly parallel decoding process of the entire sparse matrix. The proposed sparse matrix is constructed by combining pruning and weight quantization. For the latest RNN models on PTB and WikiText-2 corpus, LSTM parameter storage requirement is compressed 19× using the proposed sparse matrix format compared to the baseline model. Compressed weight and indices can be reconstructed into a dense matrix fast using Viterbi encoders. Simulation results show that the proposed scheme can feed parameters to processing elements 20 % to 106 % faster than the case where the dense matrix values directly come from DRAM.

Summary:
This paper addresses the computational aspects of Viterbi-based encoding for neural networks.
In usual Viterbi codes, input messages are encoded via a convolution with a codeword, and then decoded using a trellis. Now consider a codebook with n convolutional codes, of rate 1/k. Then a vector of length n is represented by inputing a message of length k and receiving n encoded bits. Then the memory footprint (in terms of messages) is reduced by rate k/n. This is the format that will be used to encode the row indices in a matrix, with n columns.  (The value of each nonzero is stored separately.)  However, it is clear that not all messages are possible, only those in the "range space" of my codes. (This part is previous work Lee 2018.)
The "Double Viterbi" (new contribution) refers to the storage of the nonzero values themselves. A weakness of CSR and CSC (carried over to the previous work) is that since each row may have a different number of nonzeros, then finding the value of any particular nonzero requires going through the list to find the right corresponding nonzero, a sequential task. Instead, m new Viterbi decompressers are included, where each row becomes (s_1*codeword_1 + s_2*codeword2 + ...) cdot mask, and the new scalar are the results of the linear combinations of the codewords.
Pros:
 - I think the work addressed here is important, and though the details are hard to parse and the new contributions seemingly small, it is important enough for practical performance.
 - The idea is theoretically sound and interesting.
Cons:
 - My biggest issue is that there is no clear evaluation of the runtime benefit of the second Viterbi decompressor. Compressability is evaluated, but that was already present in the previous work. Therefore the novel contribution of this paper over Lee 2018 is not clearly outlined.
 - It is extremely hard to follow what exactly is going on; I believe a few illustrative examples would help make the paper much clearer; in fact the idea is not that abstract.
 - Minor grammatical mistakes (missing "a" or "the" in front of some terms, suggest proofread.)


A UNIFIED THEORY OF EARLY VISUAL REPRESENTA- TIONS FROM RETINA TO CORTEX THROUGH ANATOMI- CALLY CONSTRAINED DEEP CNNS
The vertebrate visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex (V1), typical RFs are sharply tuned to a precise orientation. There is currently no unified theory explaining these differences in representations across layers. Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and for the first time we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing. The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck. Second, we find that, for simple downstream cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortical networks. This result predicts that the retinas of small vertebrates (e.g. salamander, frog) should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli. These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.

This paper addresses questions about the representation of visual information in the retina. The authors create a deep neural network model of the visual system in which a single parameter (bandwidth between the “retina” and “visual cortex” parts) is sufficient to qualitatively reproduce retinal receptive fields observed across animals with different brain sizes, which have been hard to reconcile in the past.
This work is an innovative application of deep neural networks to a long-standing question in visual neuroscience. While I have some questions about the analyses and conclusions, I think that the paper is interesting and of high quality.
My main concern is that the authors only show single examples, without quantification, for some main results (RF structure). For example, for Fig. 2A and 2B, an orientation selectivity index should be shown for all neurons. A similar population analysis should be devised for Fig 2C, e.g. like Fig 3 in [1]
Minor comments:
1. Page 4: “These results suggest that the key constraint ... might be the dimensionality bottleneck..”: The analyses only show that the bottleneck is *sufficient* to explain the differences, but “the key constraint” also implies *necessity*. Either soften the claim or provide control experiments showing that alternative hypotheses (constraint on firing rate etc.) cannot explain this result in your model.
2. I don’t understand most of the arguments about “cell types” (e.g. Fig. 2F and elsewhere). In neuroscience, “cell types” usually refers to cells with completely different connectivity constraints, e.g. excitatory vs. inhibitory cells or somatostatin vs. parvalbumin cells. But you refer to different CNN channels as different “types”. This seems very different than the neuroscience definition. CNN channels just represent different feature maps, i.e. different receptive field shapes, but not fundamentally different connectivity patterns. Therefore, I also don’t quite understand what you are trying to show with the weight-untying experiments (Fig. 2E/F).
3. It is not clear to me what Fig. 3B and the associated paragraph are trying to show. What are the implications of the nonlinearity being due to the first or second stage?
4. Comment on Fig 3F: The center-surround RFs probably implement a whitening transform (which is linear). Whitened inputs can probably be represented more efficiently in a network trained with L2-regularization and/or SGD. This might explain why the “quasi-linear” retina improves separability later-on.
[1] Cossell, Lee, Maria Florencia Iacaruso, Dylan R. Muir, Rachael Houlton, Elie N. Sader, Ho Ko, Sonja B. Hofer, and Thomas D. Mrsic-Flogel. “Functional Organization of Excitatory Synaptic Strength in Primary Visual Cortex.” Nature 518, no. 7539 (February 19, 2015): 399–403. https://doi.org/10.1038/nature14182.


Multi-agent collaboration is required by numerous real-world problems. Although distributed setting is usually adopted by practical systems, local range communication and information aggregation still matter in fulfilling complex tasks. For multi-agent reinforcement learning, many previous studies have been dedicated to design an effective communication architecture. However, existing models usually suffer from an ossified communication structure, e.g., most of them predefine a particular communication mode by specifying a fixed time frequency and spatial scope for agents to communicate regardless of necessity. Such design is incapable of dealing with multi-agent scenarios that are capricious and complicated, especially when only partial information is available. Motivated by this, we argue that the solution is to build a spontaneous and self-organizing communication (SSoC) learning scheme. By treating the communication behaviour as an explicit action, SSoC learns to organize communication in an effective and efficient way. Particularly, it enables each agent to spontaneously decide when and who to send messages based on its observed states. In this way, a dynamic inter-agent communication channel is established in an online and self-organizing manner. The agents also learn how to adaptively aggregate the received messages and its own hidden states to execute actions. Various experiments have been conducted to demonstrate that SSoC really learns intelligent message passing among agents located far apart. With such agile communications, we observe that effective collaboration tactics emerge which have not been mastered by the compared baselines.

Summary:
This paper expands on the work on 'emergent communication' with 2 innovations:
- The architecture has a separate 'message channel' that processes the incoming and outgoing messages mostly independently of the hidden state of the agent. There are also dedicated architecture elements for the interaction between the hidden state and the message stream.
- The outgoing message is gated with a 'speak' action: only when the agent takes the speak-action at time step t is a message sent out at timestep t+1.
Comments for improvement:
-The paper proposes a rather complicated architecture, with many moving part. In the paper's current form it is extremely hard to see which part of this architecture contribute to the success of the method. A set of ablation studies on the different components would indeed be very helpful.
-Using the word 'thought' to describe the hidden state of the agent is rather distracting.
-Equation (1): This just seems to be the policy gradient term for a factorised action space across 'environment action' and 'communication action'. The only obvious difference is that the policy here is shown to condition on the state representation s_t, rather than on the input. Is that intended?
-The paper suffers from a lot of undefined notation, e.g. the s_t above. Please clarify.
-In Figure 2b) the MCU is shown to produce the action a_t as an output. That seems like a mistake.
-Figure 4): The results seem to be extremely unstable, which is a well known issue for independent learning. Recent work (MADDPG, COMA) has shown that centralised critics can drastically avoid these instabilities and improve final performance. Did you compare against using a centralised critic, V(central state), rather the V(observation)? Also, using a single seed on this kind of unstable learning process renders the results highly non-conclusive.
-In Figure (5), what are the red-arrows? Do these correspond to the actual actions taken by the agents or are they simply annotations? It would be good to see how far the communication range is by comparison. Also, why is there a blob of 'communicating' agents far from the enemy?
-Are different methods in the large scale battle task trained in self-play and then pitched against other methods in a round-robin tournament after training has finished or are they trained against each other?
-In Figure 6 (a), why are average rewards changing over the course of training? I would expect this to be a zero-sum setting in self-play.
-I couldn't find any supplementary material referenced in the text for the details. Instead the paper seems to have another copy of the paper itself attached in the pdf. This makes it hard to evaluate the paper given that few details around training are provided in the main text.
Overall I am concerned that the learning method used in the paper (independent baseline) is known to be unstable and to produce poor results in the multi-agent setting (see COMA and MADDPG). This raises the concern that the communication channel is mostly useful for overcoming the issues introduced from having a decentralised critic.


Unlike traditional approaches that focus on the quantization at the network level, in this work we propose to minimize the quantization effect at the tensor level. We analyze the trade-off between quantization noise and clipping distortion in low precision networks. We identify the statistics of various tensors, and derive approximate analytical expressions for the mean-square-error degradation due to clipping. By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping. For example, just by choosing the accurate clipping values, more than 40% accuracy improvement is obtained for the quantization of VGG16-BN to 4-bits of precision. Our results have many applications for the quantization of neural networks at both training and inference time. One immediate application is for a rapid deployment of neural networks to low-precision accelerators without time-consuming fine tuning or the availability of the full datasets.

This paper derives a formula for finding the minimum and maximum clipping values for uniform quantization which minimize the square error resulting from quantization, for either a Laplace or Gaussian distribution over pre-quantized value. This seems like too small a contribution to warrant a paper. I wasn't convinced that appropriate baselines were used in experiments. There were a number of statements that I believed to be technically slightly incorrect. There were also some small language problems (though these didn't hinder understanding).
more specific comments:
abstract:
"derive exact expressions" -- these expressions aren't exact. they turn out to be based on a piecewise zeroth order Taylor approximation to the density.
main paper:
"allow fit bigger networks into" -> "allow bigger network to fit into"
"that we are need" -> "that need"
"introduces an additional" -> "introduces additional"
clippig -> clipping
it's not clear a-priori that information loss is the property to minimize that maximizes performance of the quantized network.
"distributions of tensors" -> "distribution of tensor elements"
this comment also applies in a number of other places, where the writing refers to the marginal distribution of values taken on by entries in a tensor as the distribution over the tensor. note that a distribution over tensors is a joint distribution over all entries in a tensor. e.g. it would capture things like eigenvalues, entry-entry covariance, rather than just marginal statistics.
"than they could have by working individually" -> "than could have been achieved by each individually"
Why the focus on small activation bit depth? I would imagine weight bit-depth was more important than activation bit depth. Especially since you're using ?32-bit? precision in the weight/activations multiplications, so activations are computed at a high bit depth anyways.
Table 1: Give absolute accuracies too! Improvement relative to what baseline?
sec 2:
sufficeint -> sufficient
\citep often used when it should instead be \citet.
"As contrast" -> "In contrast"
section 3:
uniformity -> uniformly
I don't believe the notion of p-value is being used correctly here w.r.t. the Kolmogorov-Smirnov test.
Figure 1: The mean square error should never go to 0. This suggests something is wrong. If it's just a scaling issue, consider a semilogy plot.
Figure 2: I'm unclear what baseline (no clipping) refers to in terms of clipping values. For uniform quantization there needs to be some min and max value.


EPISODIC CURIOSITY THROUGH REACHABILITY
Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself -thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward -making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory -which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work -when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in VizDoom, DMLab and MuJoCo. In navigational tasks from VizDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only. The code is available at https://github.com/google-research/episodic-curiosity.

The main idea of this paper is to propose a heuristic method for exploration in deep reinforcement learning. The work is fairly innovative in its approach, where an episodic memory is used to store agent’s observations while rewarding the agent for reaching novel observations not yet stored in memory. The novelty here is determined by a pre-trained network that computes the within k-step-reachability of current observation to the observations stored in memory. The method is quite simple but promising and can be easily integrated with any RL algorithm.
They test their method on a pair of 3D environments, VizDoom and DMLab. The experiments are well executed and analysed.
Positives:
-	They do a rigorous analysis of parameters, and explicitly count the pre-training interactions with the environment in their learning curves.
-	This method does not hurt when dense environmental rewards are present.
-	The memory buffer is smaller than the episode length, which avoids trivial solutions.
-	The idea of having a discriminator assess distance between states is interesting.
Questions and critics:
-	The tasks explored in this paper are all navigation based tasks, would this method also apply equally successfully to non-navigation domains such as manipulation?
-	My main concern is that the pre-training of the embedding and comparator networks directly depends on how good the random exploration policy is that collects the data. In navigation domains it makes sense that the random policy could cover the space fairly well, however, this will not be the case for more complex tasks involving more complex dynamics.
-	It was surprising to me that the choice of k does not seem to be that important. As it implicitly defines what “novelty” means for an environment, I would have expected that its value should be calibrated better. Could that be a function of the navigation tasks considered?
-	The DMLab results are not great or comparable to the state-of-the-art methods, which may hinder interpreting how good the policies really are. This was perhaps a conscious choice given they are only interested in early training results, but that seems like a confound.
-	The architecture does not include an RNN which makes certain things very surprising even though they shouldn't (e.g. firing, or moving around a corner, are specifically surprising for ICM) as they cannot be learnt, but perhaps if they had an RNN in the architecture these would be easy to explain? Would be interesting to see what are the authors thoughts on this (apart from their computational complexity argument they mention)?
-	Having the memory contain only information about the current episode with no information transfer between episodes seems a bit strange to me, I would like to hear the motivation behind this?
-	The fact that the memory is reset between episodes, and that the buffer is small, can mean that effectively the method implements some sort of complex pseudo count over meta-states per episode?
-	The embedding network is only trained during the pre-training phase and frozen during the RL task. This sounds a bit limiting to me: what if the agent starts exploring part of the space that was not covered during pre-training? Obviously this could lead to collapses when allowing to fine-tune it, but I feel this is rather restrictive. Again, I feel that the choice of navigation tasks did not magnify this problem, which would arise more in harder exploration tasks.
-	I think that alluding that their method is similar to babies’ behaviour in their cradle is stretched at best and not a constructive way to motivate their work…
-	In Figure 6 and 7, all individual curves from each seed run are shown, which is a bit distracting. Perhaps showing the mean and std would be a cleaner and easier-to-interpret way to report these results?
Overall, it is a simple and interesting idea and seems quite easy to implement. However, everything is highly dependent on how varying the environment is, how bad the exploration policy used for pre-training is, how good the embeddings are once frozen, and how k, action repeat and memory buffer size interact. Given that the experiments are all navigation based, it makes it hard for me to assess whether this method can work as well in other domains with harder exploration setups.


We present a novel approach for adaptively selecting the learning rate in gradient descent methods. Specifically, we impose a regularization term on the learning rate via a generalized distance, and cast the joint updating process of the parameter and the learning rate into a maxmin problem. Some existing schemes such as AdaGrad (diagonal version) and WNGrad can be rederived from our approach. Based on our approach, the updating rules for the learning rate do not rely on the smoothness constant of optimization problems and are robust to the initial learning rate. We theoretically analyze our approach in full batch and online learning settings, which achieves comparable performances with other first-order gradientbased algorithms in terms of accuracy as well as convergence rate.

This paper presents a method for adaptively tuning the learning rate in gradient descent methods. The authors consider the formulation of each gradient descent update as a quadratic minimization problem and they propose adding a phi-divergence between the learning rate that would be used and an auxiliary vector. The authors also propose adding a maximization over all learning rates in the update.
The authors study an important problem and propose a novel method. The algorithms suggested by the author are also relatively clear, and it is great that the paper presents both theoretical results as well as numerical experiments.
On the other hand, I didn't find the main idea of hyper-regularization to be well-justified. It is not clear why adding an additional regularization term for the learning rate makes sense , and it is even less clear why this should be presented as a maxmin problem. This can make the update step much more complicated and is probably why the authors also propose a simpler alternating optimization algorithm as an alternative. Unfortunately, the authors do not discuss how this alternating optimization problem relates to the original one, and the theoretical guarantees are only presented for the original algorithm. The authors also do not justify the choice of phi-divergence as the regularizer for the learning rate. The theoretical guarantees in the paper also do not suggest that the algorithm presented in the paper is better than existing state-of-the-art methods, even in specific situations (i.e. the regret bounds don't appear better than the AdaGrad regret bounds). Moreover, without tests for statistical significance, I also didn't find the experimental results sufficiently compelling.
Specific comments and questions:
1) Page 3: Equation (4): The paper would be stronger if the authors motivated why the regularization should be posed as an outer maximization.
2) Page 3: "we use the \phi-divergence as our hyper-regularization". Why is this a good choice of reuglarizer?
3) Page 3: "only a few extra calculations are required for each step". This is a misleading comment, because the maximization can be hard when phi is complicated, even if the problem splits across dimensions.
4) Page 4: "The solution of problem (5) is the same as (7) in unconstrained case". You should provide a reference for this statement as well as discuss the specific assumptions on the objective that allow you to arrive at this claim.
5) Page 4: "while the solution of (7) is more difficult to get. Thus, we choose (5) as our basic problem". This seems like a very bad motivation for choosing the maxmin formulation. For instance, the problem would be even simpler if  you didn't include this extra phi-divergence at all.
6) Page 4: "Although setting \eta-t=\beta_t is our main focus...". Why is smoothness in the learning rate a good property?
7) Page 5: Equation (11). How do these iterates relate to the ones in equation (5) (e.g. when do they coincide, if ever)?
8) Page 5: "influence the efficient of our algorithms." Grammatical error.
9) Page 6. "our algorithms are robust to the choice of initial learning rates and do not rely on the Lipschitz constant or smoothness constant". I'm not sure why this is a valuable property, since AdaGrad doesn't rely on these parameters either.
10) Page 6: Theorems 6 and 7. How do these results depend on alpha and \beta_t? This paper would be much stronger if the bounds depend on \phi more clearly and if the authors were able to show that there exist choices of phi that make this algorithm better than existing methods.
11) Page 6: Theorem 7: The dependence on G in the regret bound actually makes this worse than the AdaGrad regret bound.
12) Page 7: "KL_devergence". Typo.
13) Page 7: "different update rules were compared in advance to select the specific one for any phi divergence in the following experiments." What does this mean exactly? How much of a difference does the choice of update rule make?
14) Page 7: "growth clipping is applied to all algorithms in our framework". Why is this necessary, and how does it affect the theoretical results?
14) Page 7-8: Figures 1, 2, and 3. It's hard to interpret the significance of these results without error bars.


LAYOUTGAN: GENERATING GRAPHIC LAYOUTS WITH WIREFRAME DISCRIMINATORS
Layout is important for graphic design and scene generation. We propose a novel Generative Adversarial Network, called LayoutGAN, that synthesizes layouts by modeling geometric relations of different types of 2D elements. The generator of LayoutGAN takes as input a set of randomly-placed 2D graphic elements and uses self-attention modules to refine their labels and geometric parameters jointly to produce a realistic layout. Accurate alignment is critical for good layouts. We thus propose a novel differentiable wireframe rendering layer that maps the generated layout to a wireframe image, upon which a CNN-based discriminator is used to optimize the layouts in image space. We validate the effectiveness of Layout-GAN in various experiments including MNIST digit generation, document layout generation, clipart abstract scene generation and tangram graphic design.

Summary: This paper presents a novel GAN framework for generating graphic layouts which consists of a set of graphic elements which are geometrically and semantically related. The generator learns a function that maps input layout ( a random set of graphic elements denoted by their classes probabilities and and geometric parameters) and outputs the new contextually refined layout. The paper also explores two choices of discriminators: (1) relation based discriminator which directly extracts the relations among different graphic elements in the parameter space, and (2) wireframe rendering discriminator which maps graphic elements to 2D wireframe images using a differentiable layer followed by a CNN for learning the discriminator. The novel GAN framework is evaluated on several datasets such as MNIST, document layout comparison and clipart abstract scene generation
Pros:
- The paper is trying to solve an interesting problem of layout generation. While a large body of work has focussed on pixel generation, this paper focuses on graphic layouts which can have a wide range of practical applications.
- The paper presents a novel architecture by proposing a generator that outputs a graphic layout consisting of class probabilities and polygon keypoints. They also propose a novel discriminator consisting of a differentiable layer that takes the parameters of the output layout and generates a rasterized image representing the wireframe. This is quite neat as it allows to utilize a CNN for learning a discriminator for real / fake prediction.
- Qualitative results are shown on a wide variety of datasets - from MNIST to clipart scene generation and tangram graphic design generation. I found the clipart scene and tangram graphic design generation experiments quite neat.
Cons:
- While the paper presents a few qualitative results, the paper is missing any form of quantitative or human evaluation on clip-art scene generation or tangram graphic design generation.
- The paper also doesn’t report results on simple baselines for generating graphic layouts. Why not have a simple regression based baseline for predicting polygon parameters? Or compare with the approach mentioned in [1]
- Even for generating MNIST digits, the paper doesn’t report numbers on previous methods used for MNIST digit generation.
Interestingly, only figure 4 shows results from a traditional GAN approach (DCGAN). Why not show the output on other datasets too?
Questions / Remarks:
- Why is the input to the GAN not the desired graphic elements and pose the problem as just predicting the polygon keypoints for those graphic elements. I didn’t quite understand the motivation of choosing a random set of graphic elements and their class probabilities as input.
    - How does this work for the case of clip-art generation for example? The input to the gan is a list of all graphic elements (boy, girl glasses, hat, sun and tree) or a subset of these?
    - It is also not clear what role the class probabilities are playing this formulation.
- In section 3.3.2, it’s mentioned that the target image consist of C channels assuming there are C semantic classes for each element. What do you mean by each graphic element having C semantic classes? Also in the formulation discusses in this section, there is no further mention of C. I wasn’t quite clear what the purpose of C channels is then.
- I found Figure 3b quite interesting - it would have been nice if you expanded on that experiments and the observations you made a little more.
[1] Deep Convolutional Priors for Indoor Scene Synthesis by Wang et al


More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU-or memoryconstrained devices. Model compression (also known as distillation) alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GANassisted model compression (GAN-MC) significantly improves student accuracy for expensive models such as deep neural networks and large random forests on both image and tabular datasets. Building on these results, we propose a comprehensive metric-the Compression Score-to evaluate the quality of synthetic datasets based on their induced model compression performance. The Compression Score captures both data diversity and discriminability, and we illustrate its benefits over the popular Inception Score in the context of image classification.

Summary:
The paper proposes an approach for improving standard techniques for model compression, i.e. compressing a big model (teacher) in a smaller and more computationally efficient one (student), using data generated by a conditional GAN (cGAN). The paper suggests that the standard practice of training the student to imitate the behavior of the teacher *on the same training data* that the teacher was trained on is problematic and can lead to overfitting. Instead, the paper proposes learning a conditional GAN, which can potentially generate large amounts of realistic synthetic data, and use this data (in addition to original training data) for model compression.
Experimental results show that this idea seems to improve the performance of convnet student models on CIFAR-10 classification and random forest student models on tabular data from UCI and Kaggle.
Another contribution of the paper is to propose an evaluation metric for generative model, called the compression score. This score evaluates the quality of generated data by using it in model compression: “good” synthetic data results in a smaller gap in performance between student and teacher models.
Strengths:
-	The paper sheds a light on an interesting aspect in model compression. The idea of teaching a student model to imitate behavior of the teacher model on *new* data is interesting. In fact, it emphasizes the fact that we are mostly interested in imitating the teacher model’s capability of generalizing to new examples rather than overfitting to training examples.
-	Experiments show that for several settings (model class, architecture and datasets), using synthetic data by a cGAN can be useful in reducing the gap between student and teacher models.
-	The paper is clearly written and easy to follow.
Weaknesses:
-	The claim that reusing the same training data used for training the teacher model in model compression can lead to overfitting of student model is not very obvious and needs more experimental evidence in my opinion. One way to test this is to use some unseen real data (e.g. validation or a held-out part of training data) for model compression, and showing that it can indeed help in improving student performance.
-	The claim that cGAN can generate “infinite” amount of realistic data is too strong. In light of some well-known problems of GANs such as mode collapse [2] and low-support learned distributions [1], this assumption seems unrealistic. In fact, it is not too obvious how synthetic data by a generative model learned on *same training data as the teacher* can provide any additional information to real data.
-	While the idea of the proposed evaluation metric seems interesting, I believe it is not very practical, because:
1.	It is computationally intensive (requires training a model from scratch on fake data)
2.	It relies on performance of the compression mechanism, which might also have some idiosyncrasies that prefer some features in synthetic data which do not necessarily correspond to quality of generated data.
Questions/Suggestions:
-	In addition to using held-out real data for model compression as suggested above, a useful baseline could be using standard data-augmentation techniques in model compression.
-	What would happen if a student model is very small and cannot possibly overfit training data? Would using synthetic data be still useful there?
-	I am actually confused about a claim made when presenting compression score in Section 5. The paper claims that the best compression score is 1 (training student model on real data), while the paper shows that in fact, good synthetic data should produce *better* accuracy than using real data. I would appreciate if authors can clarify this point.
Overall recommendation:
While the paper presents an interesting problem in model compression, I’m leaning towards rejecting the paper because of the weaknesses mentioned above. That being said, I am happy to reconsider my decision if there is any misunderstanding on my part.
References:
[1] Arora, Sanjeev, and Yi Zhang. "Do GANs actually learn the distribution? an empirical study." arXiv preprint arXiv:1706.08224 (2017).
[2] Goodfellow, Ian. "NIPS 2016 tutorial: Generative adversarial networks." arXiv preprint arXiv:1701.00160 (2016).
-----
Updated score and posted a comment to author response.


Data augmentation is a technique to reduce overfitting and to improve generalization by increasing the number of labeled data samples by performing label preserving transformations; however, it is currently conducted in a trial and error manner. A composition of predefined transformations, such as rotation, scaling and cropping, is performed on training samples, and its effect on performance over test samples can only be empirically evaluated and cannot be predicted. This paper considers an influence function which predicts how generalization is affected by a particular augmented training sample in terms of validation loss. The influence function provides an approximation of the change in validation loss without comparing the performance which includes and excludes the sample in the training process. A differentiable augmentation model that generalizes the conventional composition of predefined transformations is also proposed. The differentiable augmentation model and reformulation of the influence function allow the parameters of the augmented model to be directly updated by backpropagation to minimize the validation loss. The experimental results show that the proposed method provides better generalization over conventional data augmentation methods.

[Summary]
This paper proposes a differentiable framework to learn to augment data for image classification. In particular, it uses spatial transformer and GANs as parametric data augmenters, and it formulates the validation set loss with respect to the data augmenter in a differentiable manner.
[Pros]
1.	The proposed method does not require many trials of model training under different training data, and it learns the data augmentation directly using the final classification objective.
2.	It is inspiring to extend the differentiable form of influence function across the training and validation set and then across the original and augmented data. This paper also makes use of the most recent related advance to enable stochastic learning. The theory of the paper is nice.
3.	The experimental results on MNIST (with less labeled data) and Cifar-10 are encouraging.
[Cons]
1.	Experimental results can be stronger. Especially when compared to Ratner et al., this proposed method results in marginal performance gain. Given that Ratner et al.’s method trained the data augmentation module without supervision, the supervised learning in this paper does not show strong results. In addition, the paper did not report results on a more practical dataset (such ImageNet and Places). Even for Cifar-10, the reported numbers are away from the state-of-the-art. It is important to show the practical significance of the proposed method.
2.	Data augmentation is naturally expected to be random, but the proposed method seems to learn a deterministic parameter for the augmenting transformation, which looks unnatural and limited. (Please clarify if I missed anything.)
3.	The proposed method requires a parametric model (e.g, STN, GAN). However, differentiable parametric models are not always easy to design. This probably can be the biggest obstacle to apply the proposed method widely.
Overall, the proposed method is very interesting. However, the experimental results are limited, and more discussions are needed.


LEARNING TO LEARN WITHOUT FORGETTING BY MAXIMIZING TRANSFER AND MINIMIZING INTERFERENCE
Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. 1  We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.

The transfer/ interference perspective of lifelong learning is well motivated, and combining the meta-learning literature with the continual learning literature (applying reptile twice), even if seems obvious, wasn't explored before. In addition, this paper shows that a lot of gain can be obtained if one uses more randomized and representative memory (reservoir sampling). However, I'm not entirely convinced with the technical contributions and the analysis provided to support the claims in the paper, good enough for me to accept it in its current form. Please find below my concerns and I'm more than happy to change my mind if the answers are convincing.
Main concerns:
1) The trade-off between transfer and interference, which is one of the main contributions of this paper, has recently been pointed out by [1,2]. GEM[1] talks about it in terms of forward transfer and RWalk[2] in terms of "intransigence". Please clarify how "transfer" is different from these. A clear distinction will strengthen the contribution, otherwise, it seems like the paper talks about the same concepts with different terminologies, which will increase confusion in the literature.
2) Provide intuitions about equations (1) and (2). Also, why is this assumption correct in the case of "incremental learning" where the loss surface itself is changing for new tasks?
3) The paper mentions that the performance for the current task isn't an issue, which to me isn't that obvious as if the evaluation setting is "single-head [2]" then the performance on current task becomes an issue as we move forwards over tasks because of the rigidity of the network to learn new tasks. Please clarify.
4) In eq (4), the second sample (j) is also from the same dataset for which the loss is being minimized. Intuitively it makes sense to not to optimize loss for L(xj, yj) in order to enforce transfer. Please clarify.
5) Since the claim is to improve the "transfer-interference" trade-off, how can we verify this just using accuracy? Any metric to quantify these? What about forgetting and forward transfer measures as discussed in [1,2]. Without these, its hard to say what exactly the algorithm is buying.
6) Why there isn't any result showing MER without reservoir sampling. Also, please comment on the computational efficiency of the method (which is crucial for online learning), as it seems to be very slow.
7)The supervised learning experiments are only shown on the MNIST. Maybe, at least show on CONV-NET/ RESNET (CIFAR etc).
8) It is not clear from where the gains are coming. Do the ablation where instead of using two loops of reptile you use one loop.
Minor:
=======
1) In the abstract, please clarify what you mean by "future gradient". Is it gradient over "unseen" task, or "unseen" data point of the same task. It's clear after reading the manuscript, but takes a while to reach that stage.
2) Please clarify the difference between stationary and non-stationary distribution, or at least cite a paper with the proper definition.
3) Please define the problem precisely. Like a mathematical problem definition is missing which makes it hard to follow the paper. Clarify the evaluation setting (multi/single head etc [2])
4) No citation provided for "reservoir sampling" which is an important ingredient of this entire algorithm.
5) Please mention appendix sections as well when referred to appendix.
6) Provide citations for "meta-learning" in section 1.
[1] GEM: Gradient episodic memory for continual learning, NIPS17.
[2] RWalk: Riemannian walk for incremental learning: Understanding forgetting and intransigence, ECCV2018.