Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes
We introduce a new formulation of the Hidden Parameter Markov Decision Process (HiP-MDP), a framework for modeling families of related tasks using lowdimensional latent embeddings. Our new framework correctly models the joint uncertainty in the latent parameters and the state space. We also replace the original Gaussian Process-based model with a Bayesian Neural Network, enabling more scalable inference. Thus, we expand the scope of the HiP-MDP to applications with higher dimensions and more complex dynamics.

SUMMARY
The authors consider the problem of learning multiple related tasks. It is assumed each tasks can be described by a low-dimensional vector. A multi-task neural network model can than be learned by treating this low-dimensional vector as additional input (in addition to state and action). During optimization, both these low-dimensional inputs and the network weights are optimized. The network weights are shared over all tasks. This procedure allows a new task to be modeled very quickly. Subsequently, 'virtual samples' taken from the model can be used to train a policy (using DDQN). The method is evaluated on three different task families.
REACTION TO AUTHOR'S REPLY
Thanks for the response. It cleared up many questions I had. I changed my rating accordingly. I think it would be good to show the standard deviation over independently trained models to show the variance induced (or absence of such variance) by the (random) choice of initial tasks and possibly the model training procedure. In the toy task, it seems there is always one 'red' and one 'blue' tasks in the initial data. Does that mean the initial tasks are hand-picked (rather than randomly drawn?).
TECHNICAL QUALITY
The used approach seems sensible for the task at hand. The development of the method section seems mostly unproblematic. Two issues / suggestions are 1) when tuning the model, shouldn't the update for the network weight take samples from the global buffer into account to avoid catastrophic forgetting for the next task? 2) I'm not sure why you need prioritization for the model update, if you are looking to minimize the mean squared error wouldn't you want to see all samples equally often?
Support for the method is given by experimental evaluation. The method is mostly compared to ablated versions, and not to alternatives from the literature (although one of the methods is supposed to be analogous to the method in [11], and there might not be many other appropriate methods to compare to - would a comparison to [2] be possible?). In any case, the selection of methods to compare to includes different aspects, such as model-free vs model based, and independent models vs. multitask vs. single model for all tasks. It is not clear whether multiple independent runs were performed for each methods. Please specify this, and specify what the error bars represent (standard deviation or standard error, one or two deviations, spread between roll-outs or between independent trials, how many trials).
Dependent on the experiment, the proposed method seems at least as good as most baselines, and (at least numerically) better than all baselines on the most challenging task. The additional material in the appendix also shows the runtime to be much faster compared to the GP-based method in [11].
NOVELTY
The general framework is relatively close to [11]. However, the use of neural networks and the possibility to model non-linear dependence and covariance between states and hidden parameters seems like an important and interesting contribution.
SIGNIFICANCE AND RELEVANCE
Multi-task and transfer learning seem important components for RL approaches in application domains where rapid adaptation or sample efficiency are important. The proposed method is limited to domains where the difference between tasks can be captured in a low dimensional vector.
CLARITY
There are a couple of issues with the paper's clarity. If found Figure (1) hard to understand, given its early position in the text and the absense of axis labels. I also found Algorithm 1 hard to understand. The 'real' roll-out only seemed to be used in the first episode, what is done with the data after the first episode (apart from storing in global memory for next task?). Maybe split move entire if (i=0) out of the for loop (and remove the condition), than the forloop can just contain (15) if I'm not mistaken.
The description of the experiments is missing too many details. I couldn't find in either appendix or paper how many source instances the method was trained on before evaulation on the target task for the second or third experiment. I also couldn't find how many times you independently trained the model or what error bars represent. Two source tasks for the toy tasks seems very little given that you have differences between instances across multiple axis - with just two examples would it only learn 'red' vs 'blue' since this is the most important difference?
Furthermore there are quite a few spelling errors and similar issues in the text. I'll mention some of them below under minor comments.

MINOR COMMENTS
- Typesetting: subscripts GP and BNN are typeset as products of single-letter variables
- You might want to explain the subscripts k,b,a,d in (1) briefly.
- 79: model -> to model
- since your model's epsilon distribution is not dependent on the current state, I don't think this could model hereskedasticyity (line 31). If so, please explain. (Uncertainty in weights lead to different predictive uncertainty in parts of the state-space, like a regurlar GP. But, like a regular GP this doesn't necessarily mean you model different 'noise distribution' or risk in different part of your state space )
- 'bounded number of parameters' seems somewhat ambiguous (do you mean a finite-dimensional vector, or finitely many values for this parameter?).
- 81: GP's don't really care about high-d input representation (linear scaling in input dimensionality), tractability issues mainly concern number of samples (cubic scaling for naive implementations).
- 92-93: are delta_x and delta_y really the same weather a=E or a=W as it appears from this equation?
- 100: a linear of function -> a linear function
- 114: Sec ??
- 123: is develop -> is to develop
- 169 from which learn a policy -> please rephrase
- 174 on trained two previous -> please rephrase
- 188 "of average model"-> of the average model
- 198 Appendix ??
- 230 - I'm not sure if embedded is the right word here
- 247-249 - it would be interesting to see a comparison
- 255 - "our work"
- references: It would be good to go over all of these ones as there are many small issues. E.g. the IEEE conferences mention the year and 'IEEE' twice. Names like 'Bayesian' or 'Markov' and abbreviations like STI, HIV, POMDP, CSPBVI should be capitalized. Q-learning too. For [21] I would cite the published (ICLR) version. Publication venue is missing for [35].
- appendices: just wanted to make sure you really meant exp(-10), not 1E-10. Does the prior need to be so small? Why? What do you mean when you say the prior variance changes over training? I think with |w| you mean the number of elements, but you used notation of a norm (w is a vector, not a set).


One-Shot Imitation Learning
Imitation learning has been commonly applied to solve different tasks in isolation. This usually requires either careful feature engineering, or a significant number of samples. This is far from what we desire: ideally, robots should be able to learn from very few demonstrations of any given task, and instantly generalize to new situations of the same task, without requiring task-specific engineering. In this paper, we propose a meta-learning framework for achieving such capability, which we call one-shot imitation learning. Specifically, we consider the setting where there is a very large (maybe infinite) set of tasks, and each task has many instantiations. For example, a task could be to stack all blocks on a table into a single tower, another task could be to place all blocks on a table into two-block towers, etc. In each case, different instances of the task would consist of different sets of blocks with different initial states. At training time, our algorithm is presented with pairs of demonstrations for a subset of all tasks. A neural net is trained such that when it takes as input the first demonstration demonstration and a state sampled from the second demonstration, it should predict the action corresponding to the sampled state. At test time, a full demonstration of a single instance of a new task is presented, and the neural net is expected to perform well on new instances of this new task. Our experiments show that the use of soft attention allows the model to generalize to conditions and tasks unseen in the training data. We anticipate that by training this model on a much greater variety of tasks and settings, we will obtain a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks.

Summary
---
Complex and useful robotic manipulation tasks are difficult because of the difficulty of manipulation itself, but also because it's difficult to communicate the intent of a task. Both of these problems can be alleviated through the use of imitation learning, but in order for this to be practical the learner must be able to generalize from few examples. This paper presents an architecture inspired by recent work in meta learning which generalizes manipulation of a robot arm from a single task demonstration; i.e., it does one-shot imitation learning.
The network is something like a seq2seq model that uses multiple attention mechanisms in the style of "Neural Machine Translation by Jointly Learning to Align and Translate".  There is a demonstration network, a context network and a manipulation network.  The first two produce a context embedding, which is fed to a simple MLP (the manipulation network) that produces an action that tell the arm how to move the blocks.  This context embedding encodes the demonstration sequence of states and the current state of arm in a way that selects only relevant blocks at relevant times from the demonstration.
More specifically, the demonstration network takes in a sequence (hundreds of time steps long) of block positions and gripper states (open or closed) and embeds the sequence through a stack of residual 1d convolution block. At each layer in the stack and at each time step, an attention mechanism allows that layer to compare its blocks to a subset of relevant blocks.  This sequence embedding is fed into the context network which uses two more attention mechanisms to first select the relevant period of the demonstration to the current state and then attend to relevant blocks of the demonstration.  None of the aforementioned interpretations are enforced via supervision, but they are supported by manual inspection of attention distributions.
The network is trained to build block towers that mimic those built in a demonstration in the final arrangement of blocks where the demonstration and test instance start from initially different configuration. DAGGER and Behavior Cloning (BC) are used to train the above architecture and performance is evaluated by measuring whether the correct configuration of blocks is produced or not. DAGGER and BC perform about the same. Other findings:
* The largest difference in performance is due to increased difficulty. The more blocks that need to be stacked, the worse the desired state is replicated.
* The method generalizes well to new tasks (number and height of towers), not just instances (labelings and positions of blocks).
* The final demonstration state is enough to specify a task, so a strong baseline encodes only this state but performs significantly worse than the proposed model.
Strengths
---
This is a very nice paper. It proposes a new model for a well motivated problem based on a number of recent directions in neural net design. In particular, this proposes a novel combination of elements from few-shot learning, (esp meta learning approaches like Matching Networks), imitation learning, transfer learning, and attention models. Furthermore, some attention parts of the model (at least for examples provided) seem cleanly interpretable, as shown in figure 4.
Comments/Suggestions/Weaknesses
---
There aren't any major weaknesses, but there are some additional questions that could be answered and the presentation might be improved a bit.
* More details about the hard-coded demonstration policy should be included. Were different versions of the hard-coded policy tried? How human-like is the hard-coded policy (e.g., how a human would demonstrate for Baxter)? Does the model generalize from any working policy? What about a policy which spends most of its time doing irrelevant or intentionally misleading manipulations? Can a demonstration task be input in a higher level language like the one used throughout the paper (e.g., at line 129)?
* How does this setting relate to question answering or visual question answering?
* How does the model perform on the same train data it's seen already? How much does it overfit?
* How hard is it to find intuitive attention examples as in figure 4?
* The model is somewhat complicated and its presentation in section 4 requires careful reading, perhaps with reference to the supplement. If possible, try to improve this presentation. Replacing some of the natural language description with notation and adding breakout diagrams showing the attention mechanisms might help.
* The related works section would be better understood knowing how the model works, so it should be presented later.


Learning Multiple Tasks with Multilinear Relationship Networks
Deep networks trained on large-scale data can learn transferable features to promote learning multiple tasks. Since deep features eventually transition from general to specific along deep networks, a fundamental problem of multi-task learning is how to exploit the task relatedness underlying parameter tensors and improve feature transferability in the multiple task-specific layers. This paper presents Multilinear Relationship Networks (MRN) that discover the task relationships based on novel tensor normal priors over parameter tensors of multiple task-specific layers in deep convolutional networks. By jointly learning transferable features and multilinear relationships of tasks and features, MRN is able to alleviate the dilemma of negativetransfer in the feature layers and under-transfer in the classifier layer. Experiments show that MRN yields state-of-the-art results on three multi-task learning datasets.

This paper proposes a solution to a very interesting problem, namely multi-task learning. The authors tackles learning task similarities. It is known to be very hard problem and it has been ignored in many problems that were proposed to solve multi-task learning. The authors proposes a Bayesian method by using tensor normal priors. The paper is well written and well connected to literature. The idea is quite novel. The authors supply a set of convincing experiments and to support their method.
Although I am quite positive about paper, I would like to get information to the following questions:
1) Although experiments are convincing, I would like to see the standard deviations (stds)for experiments? Since the dataset sizes are small then reporting stds are quite important.
2) The authors uses AlexNet as a baseline. However, AlexNet architecture is not anymore the state of the art architecture. It would be great if authors uses GoogleNet or something similar and show the improvement of their algorithm.


Wasserstein Learning of Deep Generative Point Process Models
Point processes are becoming very popular in modeling asynchronous sequential data due to their sound mathematical foundation and strength in modeling a variety of real-world phenomena. Currently, they are often characterized via intensity function which limits model's expressiveness due to unrealistic assumptions on its parametric form used in practice. Furthermore, they are learned via maximum likelihood approach which is prone to failure in multi-modal distributions of sequences. In this paper, we propose an intensity-free approach for point processes modeling that transforms nuisance processes to a target one. Furthermore, we train the model using a likelihood-free leveraging Wasserstein distance between point processes. Experiments on various synthetic and real-world data substantiate the superiority of the proposed point process model over conventional ones.

This paper proposes to perform estimation of a point process using the Wasserstein-GAN approach.
More precisely, given data that has been generated by a point process on the real line, the goal is to build a model of this point process. Instead of using maximum likelihood, the authors proposed to use WGAN.
This requires to:
- define a distance between 2 realizations of a point process
- define a family of Lipschitz functions with respect to this distance
- define a generative model which transforms "noise" into a point process
The contribution of the paper is to propose a particular way of addressing these three points and thus demonstrate how to use WGAN in this setting.
The resulting approach is compared on a variety of point processes (both synthetic and real) with maximum likelihood approaches and shown to compare favorably (especially when the underlying intensity model is unknown).
I must admit that I am not very familiar with estimation of point processes and the corresponding applications and thus cannot judge the potential impact and relevance of the proposed method. However, I feel that the adaptation of WGAN (which is becoming increasingly popular in a variety of domains) to
the estimation of point processes is not so obvious and the originality of the contribution comes from proposing a reasonable approach to do this adaptation, along with some insights regarding the implementation of the Lipschitz constraint which I find interesting.
One aspect that could be further clarified is regarding the modeling of the generator function: from the definition in equation (7) there is no guarantee
that the generated sequence t_i will be increasing. It is the case that the weights are constrained to be positive for example? or is it the case that the algorithm works even when the generated sequence is not increasing since the discriminator function would discriminate against such sequences and
thus encourage the generator to produce increasing ones?


Predictive State Recurrent Neural Networks
We present a new model, Predictive State Recurrent Neural Networks (PSRNNs), for filtering and prediction in dynamical systems. PSRNNs draw on insights from both Recurrent Neural Networks (RNNs) and Predictive State Representations (PSRs), and inherit advantages from both types of models. Like many successful RNN architectures, PSRNNs use (potentially deeply composed) bilinear transfer functions to combine information from multiple sources. We show that such bilinear functions arise naturally from state updates in Bayes filters like PSRs, in which observations can be viewed as gating belief states. We also show that PSRNNs can be learned effectively by combining Backpropogation Through Time (BPTT) with an initialization derived from a statistically consistent learning algorithm for PSRs called two-stage regression (2SR). Finally, we show that PSRNNs can be factorized using tensor decomposition, reducing model size and suggesting interesting connections to existing multiplicative architectures such as LSTMs and GRUs. We apply PSRNNs to 4 datasets, and show that we outperform several popular alternative approaches to modeling dynamical systems in all cases.

This paper extends predictive state representations (PSRs) to a multilayer RNN architecture. A training algorithm for this model works in two stages: first the model is initialized by two-stage regression, where the output of each layer serves as the input for the next layer, next the model is refined by back propagation through time. The number of parameters can be controlled by factorizing the model. Finally, the performance of this model is compared to LSTMs and GRUs on four datasets.
The proposed extension is non-trivial and seem to be useful, as demonstrated by the experimental results. However, the initialization algorithm, which is a crucial learning stage, should be described more clearly. Specifically, the relation of the estimation equations (3-5) to Heffny et al. is not straightforward and requires a few intermediate explanations. On the other hand, Section 3 can be shortened. The introduction of the normalization in Equation (1) is a bit confusing as it takes a slightly different form later on.
Two strong aspects of the paper are that initialization algorithm exploits a powerful estimation procedure and the Hilbert embedding provides a rich representation. More generally, exploiting existing approaches to learn dynamical systems is an interesting avenue of research. For example, [Haarjona etal. 2016] proposed construction of RNNs by combining neural networks with Kalman Filter computation graphs.
In the experimental section, can the authors specify what kernels have been used?
It would be interesting to elaborate on the added value of additional layers. The experiments show the added value of two-layer models with respect to a one-layer in terms of test performance measure. Is this added value achieved immediately after initialization, or only after applying BPTT? Does the second layer improve by making the features richer? Does the second stage regression learn the residuals of the first one.
Minor comments and typos:
- Figures are too small
- Line 206: delete one ‘the’
Ref:
Haarjona etal. 2016, Backprop KF: Learning Discriminative Deterministic State Estimators


Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks
Collecting large training datasets, annotated with high-quality labels, is costly and time-consuming. This paper proposes a novel framework for training deep convolutional neural networks from noisy labeled datasets that can be obtained cheaply. The problem is formulated using an undirected graphical model that represents the relationship between noisy and clean labels, trained in a semisupervised setting. In our formulation, the inference over latent clean labels is tractable and is regularized during training using auxiliary sources of information. The proposed model is applied to the image labeling problem and is shown to be effective in labeling unseen images as well as reducing label noise in training on CIFAR-10 and MS COCO datasets.

The paper proposes a method for incorporating data with noisy labels into a discriminative learning framework (assuming some cleanly labeled data is also available), by formulating a CRF to relate the clean and noisy labels. The CRF sits on top of a neural network (e.g., CNN in their experiments), and all parameters (CRF and NN) are trained jointly via an EM approach (utilizing persistent contrastive divergence). Experiments are provided indicating the model outperforms prior methods on leveraging noisy labels to perform well on classification tasks, as well as inferring clean labels for noisily-labeled datapoints.
The technique is generally elegant: the graphical model is intuitive, capturing relationships between the data, clean/noisy labels, and latent "hidden" factors that can encode structure in the noisy labels (e.g., presence of a common attribute described by many of the noisy labels), while permitting an analytically tractable factorized form. The regularization approach (pre-training a simpler unconditional generative model on the cleanly labeled data to constrain the variational approximation during the early part of training) is also reasonable and appears effective.
The main downside of the technique is the use of persistent contrastive divergence during training, which (relative to pure SGD style approaches) adds overhead and significant complexity. In particular the need to maintain per-datapoint Markov chains could require a rather involved setup for truly large scale noisy datasets with hundreds of millions to billions of datapoints (such as might be obtained from web-based or user-interaction data).
The experiments are carefully constructed and informative. I was particularly happy to see the ablation study estimating the contribution of different aspects of the model; the finding that dropping the data <-> noisy-label link improves performance makes sense in retrospect, although wasn't something I was expecting. I would have been curious to see an additional experiment using the full COCO dataset as the "clean" data, but leveraging additional Flickr data to improve over the baseline (what would be the gain?).
The paper was quite readable, but could benefit from another pass to improve language clarity in a few places. For instance, the sentence beginning on line 30 took me a while to understand.
Overall, I think the paper is a valuable contribution addressing an important topic, so I recommend acceptance.


Online Learning with Transductive Regret
We study online learning with the general notion of transductive regret, that is regret with modification rules applying to expert sequences (as opposed to single experts) that are representable by weighted finite-state transducers. We show how transductive regret generalizes existing notions of regret, including: (1) external regret; (2) internal regret; (3) swap regret; and (4) conditional swap regret. We present a general and efficient online learning algorithm for minimizing transductive regret. We further extend that to design efficient algorithms for the time-selection and sleeping expert settings. A by-product of our study is an algorithm for swap regret, which, under mild assumptions, is more efficient than existing ones, and a substantially more efficient algorithm for time selection swap regret.

The paper considers the classic setting of prediction with expert advice and proposes a new notion of regret called "transductive regret", which measures the performance of the learner against the best from a class of finite-state transducers. Precisely, a transducer will read the history of past predictions of the learner and output a distribution over the next predictions; for forming the next prediction, the learner can take advantage of the class of transducers under consideration. The resulting notion of regret generalizes a number of known settings such as external and internal regret, swap regret and conditional swap regret. The main contribution is proposing an algorithm that achieves a regret of at most sqrt(E*T), where E is bounded by the number of transitions in the considered class of transducers. The authors also extend their algorithm to other settings involving time-selection functions and sleeping experts.
The paper is very technical and seems to target a rather specialized audience. The considered setting is very general and highly abstract---which is actually my main concern about the paper. Indeed, the authors fail to motivate their very general setting appropriately: the only justification given is that the new notion generalizes the most general related regret notion so far, that of conditional swap regret. There is one specific example in Section 4.2 that was not covered by previous results, but I'm not sure I understand why this particular regret notion would be useful. Specifically, I fail to see how exactly is the learner "penalized" by having to measure its regret against a restricted set of experts after having picked another expert too many times---if anything, measuring regret against a smaller class just makes the problem *easier* for the learner. Indeed, it is trivial to achieve bounded regret in this setting: just choose "b" n times and "a" in all further rounds, thus obtaining a regret of exactly n, independently of the number of rounds T. The paper would really benefit from providing a more convincing motivating example.
The paper is easy to read for the most part, with the critical exception of Section 4.1 that describes the concept of weighted finite-state transducers (WFSTs). This part was quite confusing for me to read, specifically because it is never explicitly revealed that in the specific online learning setting, labels will correspond to experts and the states will remain merely internal variables of WFSTs. Actually, the mechanism of how a WFST operates and produces output labels is never described either---while the reader can certainly figure this out after a bit of thought, I think a few explanatory sentences would be useful to remove ambiguity. The fact that the letters E and T are so heavily overloaded does not help readability either.
The analysis techniques appear to be rather standard, with most technical tools apparently borrowed from Blum and Mansour (2007). However, this is difficult to tell as the authors do not provide any discussion in the main text about the intuition underlying the analysis or the crucial difficulties that needed to be overcome to complete the proofs. The authors also provide some computational improvements over the standard reduction of Blum and Mansour (2007) and its previous improved variants, but these improvements are, in my view, minor. Actually, at the end of Section 5, the authors claim an exponential improvement over a naive implementation, but this seems to contradict the previous discussion at the beginning of the section about the work of Khot and Ponnuswami (2008).
I would also like to call the work of Maillard and Munos (ICML 2011) into the authors' attention: they also study a similar notion of regret relative to an even more general class of functions operating over histories of actions taken by the learner. This work is definitely relevant to the current paper and the similarities/differences beg to be discussed.
Overall, while I appreciate the generality of the considered setting and the amount of technical work invested into proving the main results, I am not yet convinced about their usefulness. Indeed, I can't really see the potential impact of this work in lack of one single example where the newly proposed regret notion is more useful than others. I hope to be convinced by the author response so that I can vote for acceptance---right now, I'm leaning a little bit towards suggesting rejection.
Detailed comments
=================
052: Notation is somewhat inconsistent, e.g., the loss function l_t is sometimes in plain typeface, sometimes bold...
055: Missing parens in the displayed equation (this is a recurring error throughout the paper).
093: Is it really important to have O(\sqrt{L_T}) regret bounds for the base learner? After inspecting the proofs, requesting O(\sqrt{T}) seems to be enough. In fact, this would allow setting alpha = 1/\sqrt{T} artificially for every base learner and would make the statement of all theorems a bit simpler.
099: "t is both the number of rounds and the number of iterations"---I don't understand; I thought the number of iterations was \tau_t = poly(t)?
Comments after rebuttal
=======================
Thanks for the response. I now fully appreciate the improvement in the results of Section 5, and understand the algorithmic contribution a little better. I still wish that the paper would do a better job in describing the algorithmic and proof techniques, and also in motivating the study a little better. I'm still not entirely sold on the new motivating example provided in the response, specifically: how does such a benchmark "penalize" a learning algorithm and how is this example different from the one by Mohri and Yang (2014) used to motivate conditional swap regret?
I raise my score as a sign of my appreciation, but note again that the authors have to work quite a bit more to improve the presentation in the final version.


Unsupervised Image-to-Image Translation Networks
Unsupervised image-to-image translation aims at learning a joint distribution of images in different domains by using images from the marginal distributions in individual domains. Since there exists an infinite set of joint distributions that can arrive the given marginal distributions, one could infer nothing about the joint distribution from the marginal distributions without additional assumptions. To address the problem, we make a shared-latent space assumption and propose an unsupervised image-to-image translation framework based on Coupled GANs. We compare the proposed framework with competing approaches and present high quality image translation results on various challenging unsupervised image translation tasks, including street scene image translation, animal image translation, and face image translation. We also apply the proposed framework to domain adaptation and achieve state-of-the-art performance on benchmark datasets. Code and additional results are available in https://github.com/mingyuliutw/unit.

This paper investigated the problem of unsupervised image-to-image translation (UNIT) using deep generative models. In contrast to existing supervised methods, the proposed approach is able to learn domain-level translation without paired images from two domains. Instead, the paper proposed to learn a joint distribution (two domains) from a single shared latent variable using generative adversarial networks. Additional encoder networks are used to infer the latent code from each domain. The paper proposed to jointly train the encoder, decoder, and discriminator in an end-to-end style without paired images as supervision. Qualitative results on several image datasets demonstrate good performance of the proposed UNIT pipeline.
== Qualitative Assessment ==
The unsupervised domain transfer is a challenging task in deep representation learning and the proposed UNIT network demonstrated a promising direction based on generative models. In terms of novelty, the UNIT network can be treated as an extension of coGAN paper (reference 17 in the paper) with joint training objectives using additional encoder networks.
Overall, the proposed method is sound with sufficient details including both qualitative and quantitative analysis. I like Table 1 that interpolates and compares the existing work as subnetworks in UNIT.
Considering the fact that UNIT extends coGAN paper, I would like to see more discussions and explicitly comparisons between coGAN and UNIT in the paper. Performance-wise, there seems to be slight improvement compared to coGAN (Table 3 in the paper). However, the evaluation is done on MNIST dataset which may not be conclusive at all. Also, I wonder whether images in Figure 8 are cherry-picked? Compared to coGAN figures, the examples in Figure 8 have less visual artifacts. I encourage authors to elaborate this a little bit.
* What’s the benefit of having an encoder network for image-to-image translation?
Image-to-image translation can also be applied to coGAN paper: (1) analysis-by-synthesis optimization [1] & [2]; or (2) train a separate inference network [3] & [4]. Can you possibly comment on the visual quality by applying optimization-based method to a pre-trained coGAN? I encourage authors to elaborate this a bit and provide side-by-side comparisons in the final version of the paper.
* What’s the benefit of joint training?
As being said, one can train a separate encoder/inference network. It is not crystal clear to me why VAE type of training is beneficial. Maybe the reconstruction loss can help to stabilize the training a little bit but I don’t see much help when you add KL penalty to the loss function. Please elaborate this a little bit.
Additional comments:
* Consider style-transfer as alternative method in unsupervised image-to-image translation?
Given the proposed weight sharing architecture, I would assume the translated image at least preserves the structure of input image (they are aligned to some extent). If so, style-transfer methods (either optimization based or feed-forward approximation) can serve as baseline in most experiments.
* Just for curiosity, other than visual examination, what is the metric used when selecting models for presentation?
Though the hyperparameters have little impact as described in ablation study, I wonder how did you find the good architecture for presentation? At least quantitative evaluations can be used as metric In MNIST experiment for model selection. Since you obtained good results in most experiments, I wonder if this can be elaborated a bit.
[1] Generative Visual Manipulation on the Natural Image Manifold, Zhu et al. In ECCV 2016;
[2] Reference 29 in the paper;
[3] Picture: A Probabilistic Programming Language for Scene Perception, Kulkarni et al. In CVPR 2015;
[4] Generative Adversarial Text-to-Image Synthesis, Reed et al. In ICML 2016;


Learning A Structured Optimal Bipartite Graph for Co-Clustering
Co-clustering methods have been widely applied to document clustering and gene expression analysis. These methods make use of the duality between features and samples such that the co-occurring structure of sample and feature clusters can be extracted. In graph based co-clustering methods, a bipartite graph is constructed to depict the relation between features and samples. Most existing co-clustering methods conduct clustering on the graph achieved from the original data matrix, which doesn't have explicit cluster structure, thus they require a post-processing step to obtain the clustering results. In this paper, we propose a novel co-clustering method to learn a bipartite graph with exactly k connected components, where k is the number of clusters. The new bipartite graph learned in our model approximates the original graph but maintains an explicit cluster structure, from which we can immediately get the clustering results without post-processing. Extensive empirical results are presented to verify the effectiveness and robustness of our model.

The authors propose a new method for co-clustering. The idea is to learn a bipartite graph with exactly k connected components. This way, the clusters can be directly inferred and no further preprocessing step (like executing k-means) is necessary. After introducing their approach the authors conduct experiments on a synthetic data set as well as on four benchmark data sets.
I think that the proposed approach is interesting. However, there are some issues.
First, it is not clear to me how the synthetic data was generated. Based on the first experiment I suppose it is too trivial anyway. Why did the authors not use k-means++ instead of standard k-means (which appears in 3/5 baselines)? Why did they not compare against standard spectral clustering? How did they initialize NMF? Randomly? Why not 100 random initializations like in the other methods? Why did they not use a second or third evaluation measure like the adjusted rand index or normalized mutual information? Since the authors introduce another parameter lambda (beside k) I would like to see the clustering performance in dependence of lambda to see how critical the choice of lambda really is. I see the argument why Algorithm 2 is faster than Algorithm 1, but I would like to see how they compare within the evaluation, because I doubt that they perform exactly the same. How much faster is Algorithm 2 compared to Algorithm 1?
Second, the organization of the paper has to be improved. Figure 1 is referenced already on the first page but appears on the third page. The introduced baseline BSGP should be set as an algorithm on page two. Furthermore, the authors tend to use variables before introducing them, i.e., D_u, D_v and F in lines 61-63. Regarding the title: In what sense is the approach optimal? Isn't bipartite and structured the same in your case?
Third, the language of the paper should be improved. There are some weird sentence constructions, especially when embedded sentences are present. Furthermore, there are too many "the", singular/plural mistakes and a problem with tenses, i.e. Section 5.2 should be in present tense. In addition to grammar, the mathematical presentation should be improved. Equations are part of the sentences and hence, need punctation marks.
Due to the inadequate evaluation and the quality of the write-up, I vote for rejecting the paper.
More detailed comments:
*** Organization:
- title:   I do not get where the "optimal" comes from. Lines 33-34 say Bipartite=Structured, so why are both words within the title?
- line 35: Figure 1 should be on the top of page two
- lines 61-63 shoud appear as an algorithm
- line 61: in order to understand the matrices D_u and D_v, the reader has to jump to line 79 and then search backwards for D, which is defined in line 74..
- line 63: define D_u, D_v and F before using them..
*** Technical:
- line 35: B is used for a graph but also for the data matrix (lines 35 and 57)
- line 38: B_{ij} should be small (see lines 53 and 60)
- almost all equations: math is part of the text and therefore punctuation marks (.,) have to be used
- equation (2) and line 70, set Ncut, cut and assoc with \operatorname{}
- line 74: order: first introduce D, then d_{ii} and then L
- line 74: if D is a diagonal matrix, then D_u and D_v are diagonal too. Say it!
- equation (4): introduce I as the identity matrix
- lines 82-85: this paragraph is problematic. What is a discrete value? Should the entries of U and V be integer? ( I think there a just a few orthogonal matrices which are integer ) How does someone obtain a discrete solution of by running k-means on U and V?
- line 93: if P (and therefore S) should be non-negative (see Equation (9)), then you can already say it in line 93. Furthermore, isn't this an assumption you have to make? Until line 93, A and B could contain negative numbers but in the proposed approach, this is no longer possible, right?
- line 113: a sentence does not start with a variable, in this case \sigma
- line 132: get rid of (n=n_1+n_2) or add a where before
- line 149: \min
- line 150: Algorithms 1 and 2 are working on different graph laplacians. I understand that Alg.2 is faster than Alg.1 but shouldn't there be also a difference in performance? What are the results? It should be noted if Alg.1 performs exactly the same as Alg.2. However, another graph laplacian should make a difference.
- line 160: Why does NMF need a similarity graph?
- line 161: Did you tune the number of neighbors?
- line 164: Why did you not use k-means++?
- line 167: How does the performance of the proposed method vary with different values for the newly introduced parameter lambda?
- line 171: How does the performance of the proposed method vary with different values for k. Only the ground-truth k was evaluated..
- line 175: "two-dimensional matrix" Why not $n_1 \times n_2$?
- Section 5.2: Please improve the description on how the synthetic data was generated.
- line 182: if \delta is iid N(0,1) and you scale it with r greater than 0, then r*\delta is N(0,r^2)
- line 183: according to the values of r, the noise was always less than the "data"?
- Figure 2: Improve layout and caption.
- Table 1: After having 100 executions of all algorithms, what are the standard deviations or standard errors of the values within the table?
- line 183: r is not the portion of noise
- line 183: you did not do a evaluation of robustness!
- line 194: setting r=0.9 does not yield a high portion of noise. Perhaps the description of the synthetic data needs improvement.
- Table 2: NMF has to be randomly initialized as well, hence I am missing the average over 100 executions
- Table 2: What is +-4.59? One standard deviation? Standard error? Something else?
- line 219: You forgot to mention the obvious limitation of your approach: you assume the data to be non-negative!
- line 220: \ell_2-norm vs. L2-norm (line 53)
- line 224: Please elaborate the sentence "This is because high-dimensional..."
*** Language:
- line 4:  graph-based
- line 20: feature space
- line 22: distribution (singular)
- line 27: the duality [...] is
- line 33: In graph-based
- line 35: of such a bipartite graph
- line 37: The affinity [...] is
- line 38: propose, stay in present tense
- line 39: conduct
- line 40: "in this method" Which method is meant?
- line 43: graph-based
- lines 53-54: L2-norm vs. Frobenius norm, with or without -
- line 58: "view"
- line 66: In the following,
- line 72: component (singular)
- lines 76-77: "denotes" is wrong, better use "Let Z="
- line 82: Note that
- line 88: We can see from the previous section
- lines 88-90: this is a strange sentence
- line 93: we learn a matrix S that has
- line 96: If S has exactly k connected
- line 101: S, which have exactly k connected
- lines 104-105: In the next subsection
- line 132: to conduct an eigen-decomposition
- line 134: to conduct an SVD
- Algorithm 1: while not converged
- line 141: "will be hold"
- line 148: to conduct an SVD
- line 149: "Algorithm 2 is much efficient than Algorithm 1"
- Section 5.1: the section should be written in present tense (like Section 5.2) not in past tense
- line 159: Non-negative Matrix Factorization (NMF)
- line 204: data sets do not participate in experiments
- line 229: graph-based
- line 232: "we guaranteed the learned grapg to have explicitly k connected components"
- unnecessary use of "the" in lines 33, 68, 76, 77, 81, 82, 102, 104, 111, 114, 118, 119, 120, 122, 125, 129, 141, 143, 145


Learning Low-Dimensional Metrics
This paper investigates the theoretical foundations of metric learning, focused on three key questions that are not fully addressed in prior work: 1) we consider learning general low-dimensional (low-rank) metrics as well as sparse metrics; 2) we develop upper and lower (minimax) bounds on the generalization error; 3) we quantify the sample complexity of metric learning in terms of the dimension of the feature space and the dimension/rank of the underlying metric; 4) we also bound the accuracy of the learned metric relative to the underlying true generative metric. All the results involve novel mathematical approaches to the metric learning problem, and also shed new light on the special case of ordinal embedding (aka non-metric multidimensional scaling).

Summary of the paper:
The paper considers the problem of learning a low-rank / sparse and low-rank Mahalanobis distance under relative constraints dist(x_i,x_j) < dist(x_i,x_k) in the framework of regularized empirical risk minimization using trace norm / l_{1,2} norm regularization. The contributions are three theorems that provide 1) an upper bound on the estimation error of the empirical risk minimizer 2) a minimax lower bound on the error under the log loss function associated to the model, showing near-optimality of empirical risk minimization in this case 3) an upper bound on the deviation of the learned Mahalanobis distance from the true one (in terms of Frobenius norm) under the log loss function associated to the model.
Quality:
My main concern here is the close resemblance to Jain et al. (2016). Big parts of the present paper have a one-to-one correspondence to parts in that paper. Jain et al. study the problem of learning a Gram matrix G given relative constraints dist(x_i,x_j) < dist(x_i,x_k), but without being given the coordinates of the points x_i (ordinal embedding problem). The authors do a good job in explaining that the ordinal embedding problem is a special case of the considered metric learning problem (Section 2.4). However, I am wondering whether such a relationship in some sense holds the other way round too and how it could be exploited to derive the results in the present paper from the results in Jain et al. (2016) without repeating so many things: can’t we exploit the relationship G=X^T K X to learn K from G? At least in the noise-free case, if G=Y^T Y we can simply set K=L^T L, where L solves Y^T = X^T L^T?
I did not check the proofs in detail, but what I checked is correct. Also, many things can be found in Jain et al. (2016) already.
In Line 16, d_K(x_i,x_j) is defined as (x_i-x_j)^T K (x_i-x_j), but this is not a metric. To be rigorous, one has to take the square root in this definition and replace d_K(x_i,x_j) by d_K(x_i,x_j)^2 in the following.
In Corollary 2.1.1, the assumption is max_i \|x_i\|^2 = O(\sqrt{d}\log n), but in Theorem 2.1 it is max_i \|x_i\|^2 = 1. How can we have a weaker assumption in the corollary than in the theorem?
There is a number of minor things that should be corrected, see below.
Clarity:
At some points, the presentation of the paper could be improved:
Apparently, in Theorem 2.3 the set K_{\lambda,\gamma} corresponds to l_{1,2} norm regularization, but the definition from Line 106 has never been changed. That’s quite confusing.
Line 76: “a noisy indication of the triplet constraint d_{\hat{K}}(x_i,x_j) < d_{\hat{K}}(x_i,x_k)” --- in the rest of the paper, hat{K} denotes the estimate? And, if I understand correctly, we have y_t=-1 if the constraint is true. From the current formulation one would expect y_t=+1 in this case.
Originality:
I am missing a comparison to closely related results in the literature. What is the difference between Theorem 2.1 and Inequality (5) in combination with Example 6 in [3] (Bellet et al., 2015---I refer to the latest arxiv version)? The authors claim that “past generalization error bounds ... failed to quantify the precise dependence on ... dimension”. However, the dimension p only appears via log p and is not present in the lower bound anymore, so it’s unclear whether the provided upper bound really quantifies the precise dependence.
Huang et al. (2011) is cited incorrectly: it DOES consider relative constraints and does NOT use Rademacher complexities.
Significance:
In the context of metric learning, the results might be interesting, but a big part of the results (formulated in the setting of ordinal embedding --- see above) and techniques can be found already in Jain et al. (2016).
Minor points:
- Line 75, definition of T: in a triple (i,j,k), we assume j < k
- Line 88: that’s an equation and not a definition
- Line 98: if I_n is used to denote the identity, why not use 1_n?
- Line 99: XV instead of VX
- Line 100/101: also define the K-norm used in Line 126 and the ordinary norm as Euclidean norm or operator norm
- Be consistent with the names of the norm: in Line 101 you say “the mixed 1,2 norm”, in Line 144 “a mixed l_{1,2} norm”.
- In Corollary 2.1.1: shouldn’t the log n-term be squared?
- Line 190: remove the dot
- Line 212: I think “rank of the” is missing
- Typos in Line 288, Line 372, Line 306, Line 229, Line 237, Line 240, Line 259
- Line 394: “We” instead of “I”
- Line 335: for K^1 \noteq K^2


Best Response Regression
In a regression task, a predictor is given a set of instances, along with a real value for each point. Subsequently, she has to identify the value of a new instance as accurately as possible. In this work, we initiate the study of strategic predictions in machine learning. We consider a regression task tackled by two players, where the payoff of each player is the proportion of the points she predicts more accurately than the other player. We first revise the probably approximately correct learning framework to deal with the case of a duel between two predictors. We then devise an algorithm which finds a linear regression predictor that is a best response to any (not necessarily linear) regression algorithm. We show that it has linearithmic sample complexity, and polynomial time complexity when the dimension of the instances domain is fixed. We also test our approach in a high-dimensional setting, and show it significantly defeats classical regression algorithms in the prediction duel. Together, our work introduces a novel machine learning task that lends itself well to current competitive online settings, provides its theoretical foundations, and illustrates its applicability.

STRENGTHS:
On some "abstract" level, this paper pursues an interesting direction:
- I fully agree that "prediction is not done in isolation" (l24), and though this is a vague observation, it is worth pursuing this beyond established work (although I'm not an expert with a full overview over this field).
- Making explicit the tradeoff between good-often and not-bad-always in deciding on a loss function is interesting.
Overall, the writing and mathematical quality of the paper is good to very good, with some uncertainty regarding proofs though:
- Well written and structured.
- The paper is very strong in that the algorithmic and mathemtaical results are well connected (combine the VC theorems and lemmas to reach corollary 1 which shows sample bounds for their EPM algo) and cover the usual questions (consistency, runtime).
- In terms of correctness of the mathematical results, it seems that the authors know what they are doing. I only checked one proof, that of Theorem 1 (via the second proof of Lemma 5), and it seems correct although I couldn't follow all steps.
WEAKNESSES:
For me the main issue with this paper is that the relevance of the *specific* problem that they study -- maximizing the "best response" payoff (l127) on test data -- remains unclear. I don't see a substantial motivation in terms of a link to settings (real or theoretical) that are relevant:
- In which real scenarios is the objective given by the adverserial prediction accuracy they propose, in contrast to classical prediction accuracy?
- In l32-45 they pretend to give a real example but for me this is too vague. I do see that in some scenarios the loss/objective they consider (high accuracy on majority) kind of makes sense. But I imagine that such losses already have been studied, without necessarily referring to "strategic" settings. In particular, how is this related to robust statistics, Huber loss, precision, recall, etc.?
- In l50 they claim that "pershaps even in most [...] practical scenarios" predicting accurate on the majority is most important. I contradict: in many areas with safety issues such as robotics and self-driving cars (generally: control), the models are allowed to have small errors, but by no means may have large errors (imagine a self-driving car to significantly overestimate the distance to the next car in 1% of the situations).
Related to this, in my view they fall short of what they claim as their contribution in the introduction and in l79-87:
- Generally, this seems like only a very first step towards real strategic settings: in light of what they claim ("strategic predictions", l28), their setting is only partially strategic/game theoretic as the opponent doesn't behave strategically (i.e., take into account the other strategic player).
- In particular, in the experiments, it doesn't come as a complete surprise that the opponent can be outperformed w.r.t. the multi-agent payoff proposed by the authors, because the opponent simply doesn't aim at maximizing it (e.g. in the experiments he maximizes classical SE and AE).
- Related to this, in the experiments it would be interesting to see the comparison of the classical squared/absolute error on the test set as well (since this is what LSE claims to optimize).
- I agree that "prediction is not done in isolation", but I don't see the "main" contribution of showing that the "task of prediction may have strategic aspects" yet.
REMARKS:
What's "true" payoff in Table 1? I would have expected to see the test set payoff in that column. Or is it the population (complete sample) empirical payoff?
Have you looked into the work by Vapnik about teaching a learner with side information? This looks a bit similar as having your discrapency p alongside x,y.


On the Complexity of Learning Neural Networks
The stunning empirical successes of neural networks currently lack rigorous theoretical explanation. What form would such an explanation take, in the face of existing complexity-theoretic lower bounds? A first step might be to show that data generated by neural networks with a single hidden layer, smooth activation functions and benign input distributions can be learned efficiently. We demonstrate here a comprehensive lower bound ruling out this possibility: for a wide class of activation functions (including all currently used), and inputs drawn from any logconcave distribution, there is a family of one-hidden-layer functions whose output is a sum gate, that are hard to learn in a precise sense: any statistical query algorithm (which includes all known variants of stochastic gradient descent with any loss function) needs an exponential number of queries even using tolerance inversely proportional to the input dimensionality. Moreover, this hard family of functions is realizable with a small (sublinear in dimension) number of activation units in the single hidden layer. The lower bound is also robust to small perturbations of the true weights. Systematic experiments illustrate a phase transition in the training error as predicted by the analysis.

The paper presents a new information-theoretic lower bound for learning neural networks. In particular, it gives an exponential statistical query lower bound that applies even to neural nets with only one hidden layer using common activation functions and log-concave input distributions. (In fact, the results apply to a slightly more general family of data distributions.) By assuming log-concavity for the input distribution, the paper proves a lower bound for a more realistic setting than previous works with applied to discrete distributions.
To prove this SQ lower bound, the paper extends the notion of statistical dimension to regression problems and proves that a family of functions which can be represented by neural nets with one hidden layer has exponentially high statistical dimension with respect to any log-concave distribution. (I was surprised to read that statistical dimension and SQ lower bounds have not been studied for regression problems before; however, my cursory search did not turn up any previous papers that would have studied this problem.) It is not clear to me if there are any novel technical ideas in the proofs, but the idea of studying the SQ complexity of neural networks in order to obtain lower bounds for a more realistic class of nets is (as far as I can tell) novel and clever.
The specific class of functions which are used to prove the lower bound is, like in most lower bound proofs, pretty artificial and different from functions one would encounter in practice. Therefore the most interesting consequence of the results is that reasonable-looking assumptions are not sufficient to prove upper bounds for learning NN's (as the authors explain in the abstract). On the other hand, the class of functions studied is not of practical interests, which makes me question the value of the experiments in the paper (beyond pleasing potential reviewers who dislike papers without experimental results). With the tight page limits of NIPS, it seems like the space taken up by the experimental results could have a better use.
My main problem with the paper is the clarity of the write-up. I find the paper hard to read. I expect technical lemmas to be difficult to follow, but here even the main theorem statements are hard to parse with a lot of notation and formulas and little intuitive explanation. In fact, even the "Proof ideas" section (1.4), which I expected to contain a reader-friendly, high-level intuitive explanation of the main ideas, was disappointingly difficult to follow. I had to reread some sentences several times before I understood them. Of course, it is possible that this is due to my limitations instead of the paper's, but the main conceptual ideas of the paper don't seem so complex that this complexity would make it impossible to give an easily readable overview of the theorems and proofs.
Overall, this seems to be a decent paper that many in the NIPS community would find interesting to read but in my opinion a substantial portion of it would have to be rewritten to improve its clarity.


Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization
Due to their simplicity and excellent performance, parallel asynchronous variants of stochastic gradient descent have become popular methods to solve a wide range of large-scale optimization problems on multi-core architectures. Yet, despite their practical success, support for nonsmooth objectives is still lacking, making them unsuitable for many problems of interest in machine learning, such as the Lasso, group Lasso or empirical risk minimization with convex constraints. In this work, we propose and analyze PROXASAGA, a fully asynchronous sparse method inspired by SAGA, a variance reduced incremental gradient algorithm. The proposed method is easy to implement and significantly outperforms the state of the art on several nonsmooth, large-scale problems. We prove that our method achieves a theoretical linear speedup with respect to the sequential version under assumptions on the sparsity of gradients and block-separability of the proximal term. Empirical benchmarks on a multi-core architecture illustrate practical speedups of up to 12x on a 20-core machine.

The main contribution of this paper is to offer a convergence proof for minimizing sum fi(x) + g(x) where fi(x) is smooth, and g is nonsmooth, in an asynchronous setting. The problem is well-motivated; there is indeed no known proof for this, in my knowledge.
A key aspect of this work is the block separability of the variable, which allows for some decomposition gain in the prox step (since the gradient of each fi may only effect certain indices of x.) This is an important practical assumption, as otherwise the prox causes locks, which may be unavoidable. (It would be interesting to see element-wise proxes, such as shrinkage, being decomposed as well.)
Their are two main theoretical results. Theorem 1 gives a convergance rate for proxSAGA, which is incrementally better than a previous result. Theorem 2 gives the rate for an asynchronous setting, which is more groundbreaking.
Overall I think it is a good paper, with a very nice theoretical contribution that is also practically very useful. I think it would be stronger without the sparsity assumption, but the authors also say that investigating asynchronosy without sparsity for nonsmooth functions is an open problem, so that is acknowledged.
potentially major comment:
 - Theorem 2 relies heavily on a result from leblond 2017, which assumes smooth g's. It is not clear in the proof that the parts borrowed from Leblond 2017 does not require this assumption.
minor comments about paper:
 - line 61, semicolon after "sparse"
 - line 127: "a a"
minor comments about proof (mostly theorem 1):
 - eq. (14) is backwards (x-z, not z-x)
 - line 425: do you mean x+ = x - gamma vi?
 - Lemma 7 can divide RHS by 2 (triangle inequality in line 433 is loose)
 - line 435: not the third term (4th term)
 - line 444: I am not entirely clear why EDi = I. Di has to do with problem structure, not asynchronasy?
 - eq (33) those two are identical, typo somewhere with missing Ti?
 - line 447: Not sure where that split is coming from, and the relationship between alpha i, alpha i +, and grad fi.
 - line 448: there is some mixup in the second line, a nabla f_i(x*) became nabla f_i(x).
Some of these seem pretty necessary to fix, but none of these seem fatal, especially since the result is very close to a previously proven result.
I mostly skimmed theorem 2 proof; it looks reasonable, apart from the concern of borrowing results from Leblond (assumptions on g need to be clearer)


A Decomposition of Forecast Error in Prediction Markets
We analyze sources of error in prediction market forecasts in order to bound the difference between a security's price and the ground truth it estimates. We consider cost-function-based prediction markets in which an automated market maker adjusts security prices according to the history of trade. We decompose the forecasting error into three components: sampling error, arising because traders only possess noisy estimates of ground truth; market-maker bias, resulting from the use of a particular market maker (i.e., cost function) to facilitate trade; and convergence error, arising because, at any point in time, market prices may still be in flux. Our goal is to make explicit the tradeoffs between these error components, influenced by design decisions such as the functional form of the cost function and the amount of liquidity in the market. We consider a specific model in which traders have exponential utility and exponential-family beliefs representing noisy estimates of ground truth. In this setting, sampling error vanishes as the number of traders grows, but there is a tradeoff between the other two components. We provide both upper and lower bounds on market-maker bias and convergence error, and demonstrate via numerical simulations that these bounds are tight. Our results yield new insights into the question of how to set the market's liquidity parameter and into the forecasting benefits of enforcing coherent prices across securities.

Summary:
This paper presents several results for analyzing the error of forecasts made using prediction markets. These results can be used to compare different choices for the cost function and give principled guidance about setting the liquidity parameter b when designing markets.
The first key idea is to decompose the error into three components: sampling error, market-maker bias (at equilibrium), and convergence error. Intuitively, sampling error is due to the empirical distribution of traders' beliefs not matching the unknown true distribution over states, the market-maker bias is the asymptotic error due to the market's rules influencing trader behavior, and the convergence error is the distance from the market's prices to the equilibrium prices after a fixed number of trades. To make this decomposition precise, the authors introduce two notions of equilibria for the market: market-clearing and market-maker equilibria. Intuitively, a market-clearing equilibrium is a set of security prices and allocations of securities and cash to the N traders such that no trader has incentive to buy or sell securities at the given prices (in this case, the traders need not interact via the market!). Similarly, a market-maker equilibrium is a set of prices and allocations of securities and cash to the N traders that can be achieved by trading through the market and for which no trader has incentive to buy or sell securities through the market at the current prices. With this, the sampling error is the difference between the true prices and market-clearing equilibrium prices, the market-maker bias is the difference between the market-clearing and market-maker equilibrium prices, and the convergence error is the difference between the market-maker equilibrium prices and the prices actually observed in the market after t trades. Only the market-maker bias and the convergence error depend on the design of the market, so the authors focus on understanding the influence of the market's cost function and liquidity parameter on these two quantities.
For the remainder of the paper, the authors focus on the exponential trader model introduced by Abernethy et al. (EC 2014), where each trader has an exponential utility function and their beliefs are exponential family distributions. The authors then recast prior results from Abernethy et al. (EC 2014) and Frongillo et al. (NIPS 2015) to show that in this case, the market-clearing and market-maker equilibria are unique and have prices/allocations that can be expressed as the solutions to clean optimization problems. From these results, it is possible to bound the market-maker bias by O(b), where b is the liquidity parameter of the market. Unfortunately, this upper bound is somewhat pessimistic and, since it does not have an accompanying lower bound, it can't be used to compare different cost functions for the market. The authors then present a more refined local analysis, under the assumption that the cost function satisfies a condition the authors call convex+. These results give a precise characterization the relationship between the market-clearing and market-maker equilibrium prices, up to second order terms.
To study the convergence error, the authors consider the following market protocol: on each round the market chooses a trader uniformly at random and that trader buys the bundle of securities that optimizes their utility given the current state of the market and their own cash/security allocations. Frongillo et al. (NIPS 2015) showed that under this model, the updates to the vector of security allocations to the N traders exactly corresponds to randomized block-coordinate descent on a particular potential function. The standard analysis of randomized block coordinate descent shows that the suboptimality decreases at a rate O(gamma^t) for some gamma less than one. Under the assumption that the market cost function is convex+, this paper shows both upper and lower bounds on the convergence rate, as well as gives an explicit characterization of gamma on the cost function C and the liquidity parameter b.
We are able to combine the two above results to compare different cost functions and liquidity parameters. In particular, we can compare two different cost functions by setting their liquidity parameter to ensure the market-maker bias of the two cost functions is equal, and then comparing the rates of the two resulting convergence errors. The authors apply this analysis to two cost functions, the Logarithmic Market Scoring Rule (LMSR) and Sum of Independent LMSRs (IND), showing that for the same asymptotic bias, IND requires at least 1/2 and no more than 2 times as many trades as LSMR to achieve the same error. Their theoretical findings are also verified empirically.
Comments:
The results in the paper are interesting and nontrivial. One limitation, however, is that they seem to only apply under the exponential trader model, where each trader has an exponential family belief over outcomes and a risk-averse exponential utility function. If the results are more generally applicable (e.g., maybe for other trader models we can have similar characterizations and analysis of the two market equilibria), it may be helpful to include some discussion of this in the paper. Also, is the exponential trader model inspired by the behavior of real traders in prediction markets?
In the comparison on LMSR and IND, the authors choose the liquidity parameters of the two markets so that the market-maker bias is equal. An alternative comparison that might be interesting would be to fix a time horizon of T trades and choose the market liquidity parameter to give the most accurate forecasts after T trades. It's also a bit surprising that the analysis seems to suggest that LMSR and IND are roughly comparable in terms of the number of trades required to get the same level of accuracy. This result seems to contradict the intuition that maintaining coherence between the security prices has informational advantages. Is it the case that IND requires the liquidity parameter b to be set larger and this increases the potential loss for the market?
It may be easier to understand the definition of market-clearing and market-maker equilibria if the orders of their definition were reversed. The market-clearing equilibrium definition comes immediately after the definition of the market, but does not require that the traders buy/sell securities through the market, and this took me some time to understand.
The new upper and lower bounds for randomized block coordinate descent for convex+ functions also sound interesting, and stating them in their general form might help the paper appeal to a wider audience.


SafetyNets: Verifiable Execution of Deep Neural Networks on an Untrusted Cloud
Inference using deep neural networks is often outsourced to the cloud since it is a computationally demanding task. However, this raises a fundamental issue of trust. How can a client be sure that the cloud has performed inference correctly? A lazy cloud provider might use a simpler but less accurate model to reduce its own computational load, or worse, maliciously modify the inference results sent to the client. We propose SafetyNets, a framework that enables an untrusted server (the cloud) to provide a client with a short mathematical proof of the correctness of inference tasks that they perform on behalf of the client. Specifically, SafetyNets develops and implements a specialized interactive proof (IP) protocol for verifiable execution of a class of deep neural networks, i.e., those that can be represented as arithmetic circuits. Our empirical results on three-and four-layer deep neural networks demonstrate the run-time costs of SafetyNets for both the client and server are low. SafetyNets detects any incorrect computations of the neural network by the untrusted server with high probability, while achieving state-of-the-art accuracy on the MNIST digit recognition (99.4%) and TIMIT speech recognition tasks (75.22%).

The submission is interested in allowing an entity ("the verifier") to outsource the training of neural networks to a powerful but untrusted service provider ("the prover"), while obtaining a guarantee that that the prover correctly trained the neural network. Interactive proofs (IPs) are a kind of cryptographic protocol that provide such a guarantee.
The dominant cost in neural network training lies in evaluating the network on many different inputs (at least, this was my understanding -- the submission could be a bit clearer about this point for the benefit of readers who are not experts on how neural networks are trained in practice). Hence, the submission is focused on giving a very efficient IP for evaluating a neural network on many different inputs.
The IP given in the submission applies to a rather restricted class C of neural networks: those that exclusively use quadratic activation functions and sum-pooling gates. It does not support popular activation functions like ReLU, sigmoid, or softmax, or max-pooling or stochastic pooling. The reason for this restriction is technical: existing IP techniques are well-suited mainly to "low-degree" operations, and quadratic activation functions and sum-pooling operations are low-degree.
Despite the fact that the IP applies to a rather restricted class of neural networks, the submission's experimental section states that this class of networks nonetheless achieves  state-of-the-art performance on a few benchmark learning tasks.
The submission represents a quite direct application of existing IP techniques (most notably, a highly efficient IP for matrix multiplication due to Thaler). The main technical contribution of the IP protocol lies in the observation that evaluating a neural network from the class C on many different inputs can be directly implemented via matrix multiplication (this observation exploits the linear nature of neurons), and squaring operations. This allows the authors to make use of the highly efficient IP for matrix multiplication due to Thaler.
Without this observation, the resulting IP would be much more costly, especially for the prover, as Thaler's IP for matrix multiplication is far more efficient than general IPs for circuit-checking that have appeared in the literature.
In summary, I view the main contributions of the submission as two-fold. First, the authors observe that evaluating any neural network from a certain class (described above) on many different inputs can be reduced to operations for which highly efficient IPs are already known. Second, the submission shows experimentally that although this class is restrictive, it is still powerful enough to achieve state of the art performance on some learning tasks.
Evaluation: Although the submission contains a careful and thorough application of interactive proof techniques, it has a few weaknesses that I think leave it below the bar for NIPS. First, as a rather direct application of known IP methods, the technical novelty is a bit low. Second, the class of networks supported is restrictive (though, again, the experiments demonstrate that the class performs quite well on some learning tasks).
Third, and perhaps most importantly, is an issue of motivation. The interactive proof in the submission allows the verifier to ensure that the neural network was correctly trained on a specific training set using a specific training algorithm. But people shouldn't care *how* the network was trained -- they should only care that the trained network makes accurate predictions (i.e., has low generalization error). Hence, there is an obvious alternative solution to the real problem that the submission does not mention or compare against: the prover can simply send the final trained-up network to the verifier. The verifier can check that this final network makes accurate predictions simply by evaluating it on a relatively small randomly chosen selection of labeled examples, and confirming that its error on these labeled examples is low.
Based on the reported numbers for the verifier (i.e., an 8x-80x speedup for the verifier compared to doing the full training itself with no prover) this naive solution is in many cases likely to be more efficient for the verifier than the actual solution described in the paper.


Self-Supervised Intrinsic Image Decomposition
Intrinsic decomposition from a single image is a highly challenging task, due to its inherent ambiguity and the scarcity of training data. In contrast to traditional fully supervised learning approaches, in this paper we propose learning intrinsic image decomposition by explaining the input image. Our model, the Rendered Intrinsics Network (RIN), joins together an image decomposition pipeline, which predicts reflectance, shape, and lighting conditions given a single image, with a recombination function, a learned shading model used to recompose the original input based off of intrinsic image predictions. Our network can then use unsupervised reconstruction error as an additional signal to improve its intermediate representations. This allows large-scale unlabeled data to be useful during training, and also enables transferring learned knowledge to images of unseen object categories, lighting conditions, and shapes. Extensive experiments demonstrate that our method performs well on both intrinsic image decomposition and knowledge transfer.

The paper presents an interesting approach on the intrinsic image decomposition problem: given an input rgb image, it decomposes it first into shape (normals), reflectance (albedo) and illumination (point light) using an encoder-decoder deep architecture with 3 outputs. Then there is another encoder-decoder that takes the predicted normals and light and outputs the shading of the shape. Finally, the result comes from a multiplication between the estimated reflectance (from the 1st encoder-decoder) with the estimated shading.
The idea of having a reconstruction loss to recover the input image is interesting, but I believe that is only partially employed in the paper. The network architecture still needs labeled data for the initial training. Also, in lines 164-166 the authors mention that the unlabeled data are used together with labeled so the "representations do not shift too far from those learnt". Does this mean that the unlabeled data corrupts the network in a way that the reconstruction is valid but the intermediate results are not correct? That implies that such data driven inference without taking into account the physical properties of light/material/geometry is not optimal.
I found difficult to parse the experiments and how the network is being deployed. How are the train/validation/test sets being created? For example, in 4.1 the motorbike numbers correspond to a test set of motorbikes that were not seen during the initial training (it is mentioned that the held-out dataset is airplanes)? And for decomposing a new shape into its intrinsic properties, do you directly take the output of the 3 decoders, or you have another round of training, but with including the test data.
Another important limitation is the illumination model being just a single point light source. This simple illumination model (even a spotlight) can not be applied in real images.
Regarding that, the experiments are only on the synthetic dataset proposed in the paper and there is no comparison with [13, 15] (or a discussion why it is missing). Without testing on real images or having a benchmark such as MIT intrinsics, its difficult to argue about the effectiveness of the method outside of synthetic images.
Also, Figure 3 is the output of the network or is it an illustration of what is not captured by simple lambertian shading?
There is some related work that is missing both for radiometric and learned decomposition into intrinsic properties.
Decomposition into shape/illumination/reflectance:
Single Image Multimaterial Estimation, Lombardi et al, CVPR 12
Shape and Reflectance Estimation in the Wild, Oxholm et al, PAMI 2016, ECCV12
Reflectance and Illumination Recovery in the Wild, Lombardi et al, PAMI 2016, ECCV 12
Deep Reflectance Maps, Rematas et al, CVPR 2016
Decomposing Single Images for Layered Photo Retouching, Innamorati et al, CGF 2017 (arxiv 16)
Deep Outdoor Illumination Estimation, Hold-Geoffroy et al, CVPR 2017 (arxiv 16)
Learning shading using deep learning:
Deep Shading: Convolutional Neural Networks for Screen-Space Shading, Nalbach et al CGF 2017 (arxiv 16)


A General Framework for Robust Interactive Learning *
We propose a general framework for interactively learning models, such as (binary or non-binary) classifiers, orderings/rankings of items, or clusterings of data points. Our framework is based on a generalization of Angluin's equivalence query model and Littlestone's online learning model: in each iteration, the algorithm proposes a model, and the user either accepts it or reveals a specific mistake in the proposal. The feedback is correct only with probability p > 1 2 (and adversarially incorrect with probability 1 − p), i.e., the algorithm must be able to learn in the presence of arbitrary noise. The algorithm's goal is to learn the ground truth model using few iterations. Our general framework is based on a graph representation of the models and user feedback. To be able to learn efficiently, it is sufficient that there be a graph G whose nodes are the models, and (weighted) edges capture the user feedback, with the property that if s, s * are the proposed and target models, respectively, then any (correct) user feedback s must lie on a shortest s-s * path in G. Under this one assumption, there is a natural algorithm, reminiscent of the Multiplicative Weights Update algorithm, which will efficiently learn s * even in the presence of noise in the user's feedback. From this general result, we rederive with barely any extra effort classic results on learning of classifiers and a recent result on interactive clustering; in addition, we easily obtain new interactive learning algorithms for ordering/ranking.

The paper proposes a general framework for interactive learning. In the framework, the machine learning models are represented as the nodes in a graph G and the user feedback are represented as weighted edges in G. Under the assumption of “if s, s* are the proposed and target models, then any (correct) user feedback s’ must lie on the shortest s-s* path in G”, the author showed that the Multiplicative Weights Update algorithm can efficiently learn the target model. The framework can be applied to three important machine learning tasks: ranking, clustering, and classification. The problem investigated in the paper is interesting and important. The theoretical results of the paper is convincing. However, these results are not empirically verified on real tasks. My comments on the paper are listed as follows:
Pros.
1.	A interactive learning framework which models the models and user feedback with graph. The framework can learn the target model in the presence of random noise in the user’s feedback.
2.	The framework is really general and can be applied to major machine learning tasks such as classification, clustering and ranking.
Cons.
1.	My major concern on the paper comes from the absence of empirical evaluation of the proposed framework. The major assumption of the framework is “any (correct) user feedback s’ must lie on the shortest s-s* path in G”. Also, the authors claimed that the proposed framework is robust, which can learn the correct s* even in the presence of random noise in user’s feedback. It is necessary to verify the correctness of these assumption and conclusions based on real data and real tasks. The framework is applied to the tasks of ranking, clustering, and classification, achieving the interactive ranking, interactive clustering, and interactive classification models. It is necessary to compare these derived models with state-of-the-arts models to show the effectiveness of these models. Currently, the authors only showed the theoretical results.
2.	There is no conclusion section in the paper. It is better to conclude the contributions of the paper in the last section.


Gated Recurrent Convolution Neural Network for OCR
Optical Character Recognition (OCR) aims to recognize text in natural images. Inspired by a recently proposed model for general image classification, Recurrent Convolution Neural Network (RCNN), we propose a new architecture named Gated RCNN (GRCNN) for solving this problem. Its critical component, Gated Recurrent Convolution Layer (GRCL), is constructed by adding a gate to the Recurrent Convolution Layer (RCL), the critical component of RCNN. The gate controls the context modulation in RCL and balances the feed-forward information and the recurrent information. In addition, an efficient Bidirectional Long ShortTerm Memory (BLSTM) is built for sequence modeling. The GRCNN is combined with BLSTM to recognize text in natural images. The entire GRCNN-BLSTM model can be trained end-to-end. Experiments show that the proposed model outperforms existing methods on several benchmark datasets including the IIIT-5K, Street View Text (SVT) and ICDAR.
: Illustration of using RCL with T = 2 for OCR.
A Recurrent Convolution Neural Network (RCNN) is proposed for general image classification [22], which simulates an anatomical fact that recurrent connections are ubiquitously existent in the neocortex. It is believed that the recurrent synapses that exist in the neocortex play an important role in context modulation during visual recognition. A feed-forward model can only capture the context in higher layers where units have larger receptive fields, but this information cannot modulate the units in lower layers which is responsible for recognizing smaller objects. Hence, using recurrent connections within a convolutional layer (called Recurrent Convolution Layer or RCL) can bring context information to all units in this layer. This implements a nonclassical receptive field [12,18] of a biological neuron as the effective receptive field is larger than that determined by feedforward connections. However, with increasing iterations, the size of the effective receptive field will increase unboundedly, which contradicts the biological fact. One needs a mechanism to constrain the growth of the effective receptive field.
In addition, from the viewpoint of performance enhancing, one also needs to control the context modulation of neurons in RCNN. For example, in Figure 1, it is seen that for recognizing a character, not all of the context are useful. When the network recognizes the character "h", the recurrent kernel which covers the other parts of the character is beneficial. However, when the recurrent kernel is enlarged to the parts of other characters, such as "p", the context carried by the kernel is unnecessary. Therefore, we need to weaken the signal that comes from unrelated context and combine the feed-forward information with recurrent information in a flexible way. To achieve the above goals, we introduce a gate to control the context modulation in each RCL layer, which leads to the Gated Recurrent Convolution Neural Network (GRCNN).
In addition, the recurrent neural network is adopted in the recognition of words in natural images since it is good at sequence modeling. In this work, we choose the Long Short Term Memory (LSTM) as the top layer of the proposed model, which is trained in an end-to-end fashion.

Authors added gating mechanism in Recurrent Convolutional NN that allow controlling the amount of context / recursion. In other words, it is similar to improvement from RNN to GRU but for Recurrent CNNs. Authors benchmark their model on text reading task.
Positives aspects:
- authors improved upon the state of the art results on very well research benchmark
- the proposed model combines few proved ideas into a single model and made it work.
Areas to improve:
- Lines 195-196: you state that choose 10 best models per architecture that perform well on each dataset and state the results as the median of them. It is ok when you compare different iterations of you model to make sure you found good hyperparameters. But this cannot be at the final table with the comparison to other works. You should clearly state what is your pool of models (is it coming from different hyperparameters? what parameters and what values did you choose). How many models did you try during this search? You should provide single hyper parameter + model setup with the results that works the best across all the problems in the final table. Additionally, you may provide variance of the results with the clear explanation of hyper parameter search you performed.
- did authors try to use attention not bidirectional RNN + CTC loss? In few recent works (https://arxiv.org/pdf/1603.03101.pdf, https://arxiv.org/pdf/1704.03549.pdf, https://arxiv.org/pdf/1603.03915.pdf). This should still allow for improving the results, or if didn't work, this is worth stating.
I vote for top 50% of papers as improving upon the previous state of the art results on such a benchmark is a difficult job. Authors proposed a new model for extracting image features that is potentially beneficial to other visual tasks. This is conditional on fixing the results number (to present numbers from a single model, not median) and open-sourcing the code and models to the community for reproducibility.


Variational Memory Addressing in Generative Models
Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory.
The KL(q||p) cost to identify a specific memory entry might be substantial, even though the cost of accessing a memory entry should be in the order of log |M|. Middle & Right: We combine a top-level categorical distribution p(a) and a conditional variational autoencoder with a Gaussian p(z|m).
when retrieving contents from memory during training. We compute the sampling distribution over the addresses based on a learned similarity measure between the memory contents at each address and the target. The memory contents m a at the selected address a serve as a context for a continuous latent variable z, which together with m a is used to generate the target observation. We therefore interpret memory as a non-parametric conditional mixture distribution. It is non-parametric in the sense that we can change the content and the size of the memory from one evaluation of the model to another without having to relearn the model parameters. And since the retrieved content m a is dependent on the stochastic variable a, which is part of the generative model, we can directly use it downstream to generate the observation x. These two properties set our model apart from other work on VAEs with mixture priors [9,10] aimed at unconditional density modelling. Another distinguishing feature of our approach is that we perform sampling-based variational inference on the mixing variable instead of integrating it out as is done in prior work, which is essential for scaling to a large number of memory addresses.
Most existing memory-augmented generative models use soft attention with the weights dependent on the continuous latent variable to access the memory. This does not provide clean separation between inferring the address to access in memory and the latent factors of variation that account for the variability of the observation relative to the memory contents (see Figure 1). Or, alternatively, when the attention weights depend deterministically on the encoder, the retrieved memory content can not be directly used in the decoder.
Our contributions in this paper are threefold: a) We interpret memory-read operations as conditional mixture distribution and use amortized variational inference for training; b) demonstrate that we can combine discrete memory addressing variables with continuous latent variables to build powerful models for generative few-shot learning that scale gracefully with the number of items in memory; and c) demonstrate that the KL divergence over the discrete variable a serves as a useful measure to monitor memory usage during inference and training.

This paper proposes a memory-augmented generative model that performs stochastic, discrete memory addressing, by interpreting the memory as a non-parametric conditional mixture distribution. This variational memory addressing model can combine discrete memory addressing variables with continuous latent variables, to generate samples only with few samples in the memory, which is useful for few-shot learning. The authors implement a VAE version of their model and validate it for the few-shot recognition tasks on the Omniglot dataset, on which it significantly outperforms the Generative Matching Networks, which is an existing memory-augmented network model. Further analysis shows that the proposed model accesses relevant part of the memory even with hundreds of unseen instances in the memory.
Pros:
- Performing discrete, stochastic memory addressing for memory-augmented generative model is a novel idea which makes sense. Also, the authors have done a good job in motivating why this model is superior to soft attention approach.
- The proposed variational addressing scheme is shown to work well in case of few-shot learning, even in case where existing soft-attention model fails to work.
- The proposed scheme of interpreting the memory usage with KL divergence seems useful.
Cons
-Not much, except that the experimental study only considers character data, although they are standard datasets. It would be better if the paper provides experimental results on other types of data, such as images, and compared against (or coupled with) recent generative models (such as GANs)
Overall, this is a good paper that presents a novel, working idea. It has effectively solved the problem with existing soft-attention memory addressing model, and is shown to work well for few-shot learning. Thus I vote for accepting the paper.
- Some typos:
Line 104: unconditioneal -> unconditional
Line 135: "of" is missing between context and supervised.
Line 185: eachieving -> achieving
Table 1 is missing a label "number of shots" for the numbers 1,2,3,4,5,10 and 19.