Bandit Learning with Implicit Feedback
Implicit feedback, such as user clicks, although abundant in online information service systems, does not provide substantial evidence on users' evaluation of system's output. Without proper modeling, such incomplete supervision inevitably misleads model estimation, especially in a bandit learning setting where the feedback is acquired on the fly. In this work, we perform contextual bandit learning with implicit feedback by modeling the feedback as a composition of user result examination and relevance judgment. Since users' examination behavior is unobserved, we introduce latent variables to model it. We perform Thompson sampling on top of variational Bayesian inference for arm selection and model update. Our upper regret bound analysis of the proposed algorithm proves its feasibility of learning from implicit feedback in a bandit setting; and extensive empirical evaluations on click logs collected from a major MOOC platform further demonstrate its learning effectiveness in practice.

Summary: This work considers learning user preferences using a bandit model.  The reward is not only based on the judgement of the user, but also whether the user examined the arm. That is feedback = examination * judgement In particular, if a user does not examine an arm, lack of feedback does not necessarily indicate that the user does not "like" the arm. This work uses a latent model for the (unobserved) examination of arms, and posits that the probability of positive feedback (binary) can be expressed as a product of the probability of examination (logistic) and positive feedback (logistic). The work proposes a VI approach to estimating the parameters, and then use a Thompson Sampling approach from the approximate posterior as policy. This allows them to use machinery from Russo and Van Roy to obtain regret bounds. Furthermore, simulations on synthetic and real data from a MOOC platform are provided.  Specific comments:  - The specific form of the model (product of 2 logistic regressions) is not really motivated in the problem formulation section. Why is this the relevant model as compared to other models? How can the reader tell whether this model is also relevant to their problem?  - The objective is not given before the algorithm section. It would make sense to me to state the objective in the problem formulation section.  - Will code / data be made available for others to replicate results?  - Line 169: I assume you mean argmin instead of argmax.  - The regret bounds from Russo and Van Roy require sampling from the true posterior, but this paper considers an approximation. Thus, the regret bound does not hold for the proposed algorithm. Authors could be a bit more clear about this.  - On synthetic data, the EC algorithm beats benchmarks by a wide margin, but on the MOOC data this difference is much smaller. This suggests that there might important aspects that are not modeled by the current setup. Of course one cannot expect results on synthetic data to carry over completely, but maybe the authors can comment on this difference.  - This is a more high level comment: the paper considers a model where the platform decides what action to take / show, and the user gives feedback. However, there are many scenarios where the platform provides a set of actions, of which the user takes one (e.g. a recommendation system). In the MOOC setting considered in the paper, the model in the paper is definitely the correct one, but it is not clear to me whether this generalizes to many other scenarios as well, or that in most cases the user selects actions rather than the platform. In the paper this distinction between the two viewpoints is somewhat blurred. It would be useful to hear the authors thoughts and for the work to be more precise on which settings are and are not relevant to the model presented in the paper.  - Results in figure 4 seem very noisy. What are the SEs on the performances and could variance be reduced by running more replications?  Overall: The work tackles a realistic problem, is well written and easy to follow. Theoretical results rely on standard techniques developed earlier. THe work is missing some motivation for the specific form of the model. The simulations are not wholy convincing, since the lift for the MOOC data is much smaller than that in the synthetic data. On the other hand, using real data is a big plus and proves that the approach can be used in practice.  All in all, I'm on the fence for this one and look forward to the authors response to the issues raised  -- Update after author response: I agree with other reviewers that it is interesting model and nice application of variational inference as well. I am a bit disappointed in the author's responses considering my comment re motivation of the specific model: I understand that the model is the product of two logistic functions, but would love to see more motivation on why this model over other alternatives, and/or how this model can be validated.  Overall I remain of the opinion that it is a nice paper that covers an end-to-end use case with theory and practice. On the other hand, results are not groundbreaking so my score of a marginal accept stands.


Parsimonious Quantile Regression of Financial Asset Tail Dynamics via Sequential Learning
We propose a parsimonious quantile regression framework to learn the dynamic tail behaviors of financial asset returns. Our model captures well both the timevarying characteristic and the asymmetrical heavy-tail property of financial time series. It combines the merits of a popular sequential neural network model, i.e., LSTM, with a novel parametric quantile function that we construct to represent the conditional distribution of asset returns. Our model also captures individually the serial dependences of higher moments, rather than just the volatility. Across a wide range of asset classes, the out-of-sample forecasts of conditional quantiles or VaR of our model outperform the GARCH family. Further, the proposed approach does not suffer from the issue of quantile crossing, nor does it expose to the ill-posedness comparing to the parametric probability density function approach.

The paper proposes to express the distribution of a real-valued univariate random variable as a tail deviation from the quantile function of a normal variate, called HTQF. This function consists of 4 parameters to specify the location, scale, lower excess tail, and upper excess tail. In a time series context, the paper models these quantities at time t in a time-varying fashion through an LSTM, using a window of L past observations (and higher central moments). Since the CDF of the HTQF model is monotonically increasing, the quantile crossing problem is avoided by construction. The model is trained by minimizing the set of pinball losses at the various quantiles of interest.  The paper is quite well written and easy to follow. The proposed form for the HTQF is interesting and appears to be a novel contribution worthy of a presentation at NIPS. The experimental validation on synthetic and real datasets is adequate, and the latter appears to show some improvement over well-known GARCH models. In addition to the pinball loss, it would have been helpful to provide coverage measures for the quantiles (i.e. in a properly-calibrated 0.1 quantile, 10% of the observations should fall below it; what’s the empirical quantile corresponding to the nominal model quantile? This would be akin to a Q-Q plot between the the realized quantiles and the predicted quantiles). This measure would also more adequately compare the behavior of various models across all quantiles, which are not systematically reported in the results.  Moreover, a small quibble would be that no model from the stochastic volatility family are represented, which are the other large family of volatility models in financial econometrics apart from GARCH.  The main issue that I have with the approach is that the specification of the excess tails takes a very constrained functional form, which limits its flexibility — it would be interesting to understand whether this leads to underfitting for the financial asset return series of interest. It is also obvious that this approach can significantly enhance time series modeling in many other areas beyond finance; I would be looking forward to a discussion of which fields would appear most suited to benefit from this approach.  Overall, I think that the issues reported above are fixable, and if so I would be happy to see the paper accepted.  Detailed comments: * Line 18: care ==> care about * Line 27: gaussian ==> Gaussian * Line 41: speak itself ==> speak for itself * Line 137: has ==> have * Line 236: Verity ==> Verify * Line 236: truth ==> true * Line 249: truth ==> true * Line 249: no matter it is on ==> on both * Paragraph 248-255: this analysis is sketchy and should be made more precise.


Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation
Neural Machine Translation (NMT) has achieved remarkable progress with the quick evolvement of model structures. In this paper, we propose the concept of layer-wise coordination for NMT, which explicitly coordinates the learning of hidden representations of the encoder and decoder together layer by layer, gradually from low level to high level. Specifically, we design a layer-wise attention and mixed attention mechanism, and further share the parameters of each layer between the encoder and decoder to regularize and coordinate the learning. Experiments show that combined with the state-of-the-art Transformer model, layer-wise coordination achieves improvements on three IWSLT and two WMT translation tasks. More specifically, our method achieves 34.43 and 29.01 BLEU score on WMT16 English-Romanian and WMT14 English-German tasks, outperforming the Transformer baseline.

Update after Author Feedback:  Thanks for all the extra info about speed and alternative encoder-decoders.   Original Review:  This work builds directly off of Transformer networks. They make two contributions to that kind of architecture. The first is to suggest running the encoder and decoder stacks layer by layer instead of running the encoder stack and passing information to the decoder stack. The second is to actually tie the weights of the encoder and decoder.  +:   Shows that sharing the weights between the encoder and decoder of a transformer network is promising.This is surprising and worth knowing.  Running a decoder layer right after its corresponding encoder layer processes (rather than running the next encoder layer) is also an interesting augmentation to Transformer networks. Again surprising that this contributes to better performance. If this has been done in the past, I certainly haven’t seen it in recent work.  Adding the learned source and target tokens is an interesting method for ensuring that the model knows which language the embeddings are in.  Strong improvements on NMT over the baseline Transformer network.  -:  Giving all layers of the decoder access to corresponding layers in the encoder does not seem novel, at least in the way it is phrased at various points throughout the paper. Though the original Transformer paper (Vaswani et al 2017) did in fact use only the top level outputs of their encoder, Figure 1 of (Vaswani et al 2017) actually more closely depicts the corresponding access to encoder layers. Perhaps because of that depiction, some of the follow up work, for example Gu et al. 2017, did actually use layer wise attention without considering that a major contribution of their work.  This work only shows layer wise coordination for the Transformer, not for other kinds of encoder-decoders, so its not clear that this coordination is more generally useful. To make the broader claim, I think it is necessary to show this idea is empirically fruitful for at least one of CNNs or LSTMs as well. This is mentioned as future work, but as is, this comes off more as a tweak of the Transformer than evidence of a more generally applicable idea.  You have to have 14 layers instead of 6 by doing it this way. How does this increase in layers combined with layer-wise processing affect efficiency at train/inference time?   This paper did indeed shift the way that I would try doing seq2seq tasks in the future, and the results on NMT were strong. On first read, it seemed like an incremental change to Transformers, but after looking more closely, I do think it is worth it for the community to try out this different perspective on encoder-decoder architectures. Overall 7, confidence 4. The code is said to be open sourced after reviewing, so I'm very confident this will be reproducible.


But How Does It Work in Theory? Linear SVM with Random Features
We prove that, under low noise assumptions, the support vector machine with N m random features (RFSVM) can achieve the learning rate faster than O(1/ √ m) on a training set with m samples when an optimized feature map is used. Our work extends the previous fast rate analysis of random features method from least square loss to 0-1 loss. We also show that the reweighted feature selection method, which approximates the optimized feature map, helps improve the performance of RFSVM in experiments on a synthetic data set.

The authors analyze the use of random Fourier Features (RFF) for training linear support vector machines in the RFF feature space. Their result bounds the expected risk difference of the hinge loss (i.e. the generalization error on unseen data) for a number of different scenarios. The analysis is based on a number of assumptions, on the noise distribution, on the RKHS containing the optimal classifier, and most crucially on access to the optimal feature weights by Bach 2017. The first main result is a fast rate for kernels whose spectrum decays polynomially when the Bayes classifier is in the feature space. The second result is a fast rate specifically for the Gaussian kernel when the Bayes classifier is not necessarily in the feature space, but the data is separable by some minimum distance. Since the optimal feature weights are not available in practice, the authors rely on Bach's algorithm to approximate them in their experiments. As in non-parametric regression, optimized feature weights are beneficial for the performance of the RFF SVM.  I would like to congratulate the authors, this is an excellent contribution to a worthwhile problem, and a well structured and well written paper. I have a few comments on technical aspects, presentation and experiments below. Apart from that, I am happy for the paper to get accepted. As a side note: as with most theory papers, the conference format is a bit unfortunate, as the main paper merely leaves space to state the results. I would prefer to see these articles in ML journals with a thorough review process that covers all proofs in detail (which is quite hard to do for 6 papers during NIPS reviews).   Significance. This is the first analysis of the generalization error using RFF and SVM, and as the authors point out, most of the previous analysis for RFF were based on the idea of perturbing the optimization problem and did not lead to error bounds that justified the empirical success of RFF. What is nice that the authors treat the RFF SVM as a seperate learning model, rather than a perturbed version of the kernel SVM. What is also nice is that the rates are actually fast enough (for low dimensions) to justify the empirical success of RFF SVMs that use a fewer random features than data samples. On the downside, the rates are unattainable in practice due to the intractability of the optimized feature weights, and the rates for general uniform features are not satisfactory. A second downside is that the results for the mis-specified case (as fast as they are!) are only for the Gaussian kernel. There are approaches in regression like settings, where the assumption that the data generating function does not lie in the RKHS is relaxed using interpolation spaces, e.g. Theorem 12 in [R1].  [R1] "Density Estimation in Infinite Dimensional Exponential Families", Sriperumbudur et al, https://arxiv.org/abs/1312.3516   A. Assumptions.  A1. The assumption on the data noise seems fairly standard. It would be great, however, if the authors could provide guidelines for practicioners on diagnosing when the low noise assumption is (approximately) valid and when it is not. Such diagnosis statistics are quite common in the statistics community, and they are usually based on post-processing of some kind.  A2. It is unfortunate that the results provided are only for the case of having access to the optimized feature weights, and it is even more unfortunate that using uniform features does not seem to result in fast rates in general. Of course the authors make this very clear, and Appendix E provides some additional intuition for using the uniform feature maps (some of it should be in the main text). It is unclear to me what is meant by "general feature maps" in line 237? And what is the conjecture based on? This seems a bit vague as for example Theorem 2 also managed to work around the crucial assumption that the Bayes classifier lies in the RKHS using a specialization of the used kernel.  A3. The authors are correct that analysing the reweighted feature selection algorithm by Bach 2017 is out of scope for this paper. But they will have to acknowledge, however, that this "missing link" clearly lessens the significance of the results. A quantification of the error of the approximately optimal reweighting might be combined with the presented results into an error analysis that would be extremely relevant for practicioners.  B. Comparison to the Nystrom SVM. The authors mention that they "do not find any previous work on the performance of SVM with Nystrom method on classification problems" (line 241). I presume this is regarding "theoretical analysis of the generalization error", as there is a number of papers that deals with the very matter of RFF SVMs, e.g. [R2, R3]. What I think is missing in the paper is an acknowledgement (or a reproduction) of previous experiments that were done to compare the performance of Nystrom and RFF SVMs, e.g. [R3]. This would be in particular useful to determine which method works better in which situation, or at least to contrast the two approaches from a practical perspective. These would just one or two more lines in the plots. [R2] "Scaling up kernel svm on limited resources: A low-rank linearization approach", Zhang et al, http://proceedings.mlr.press/v22/zhang12d/zhang12d.pdf [R3] "Nystrom Method vs Random Fourier Features: A Theoretical and Empirical Comparison",  Yang et al, https://papers.nips.cc/paper/4588-nystrom-method-vs-random-fourier-features-a-theoretical-and-empirical-comparison.pdf  C. High dimensional data. It is only very briefly mentioned at the end that experiments on higher-dimensional data are a bit disappointing. Placing these experiments in the Appendix is not optimal, not just because they are the only experiments on non-synthetic data. There is an interesting question here: is it the approximate feature selection algorithm or the feature weighting itself?  D. Proof outline I think an overview of the proof strategy and significance is a must for the main paper. The proof relies on some quite heavy external results, namely Theorem 3 and 4, or modified external results (Lemma 7), and it would be great to get a better feeling for what parts of the proof contain the most significant original steps. There is a number of smaller Lemmas that seem more or less straight-forward involving taking expectations and simple bounding, but there are also some heavier parts and it would be great if the authors could shed some light onto those.  I would like to thank the authors for the nice response.


Bayesian Alignments of Warped Multi-Output Gaussian Processes
We propose a novel Bayesian approach to modelling nonlinear alignments of time series based on latent shared information. We apply the method to the real-world problem of finding common structure in the sensor data of wind turbines introduced by the underlying latent and turbulent wind field. The proposed model allows for both arbitrary alignments of the inputs and non-parametric output warpings to transform the observations. This gives rise to multiple deep Gaussian process models connected via latent generating processes. We present an efficient variational approximation based on nested variational compression and show how the model can be used to extract shared information between dependent time series, recovering an interpretable functional decomposition of the learning problem. We show results for an artificial data set and real-world data of two wind turbines.

The authors propose an approach for non-linear time alignment of sequences in a shared latent space. More specifically they explain the observed time series data via a generative process: (i) first the inputs are passed through an alignment function, a, (ii) a shared latent process, f, is applied to the aligned signal, and, (iii) a warping function is responsible to introduce output specific attributes to the observe the final signal. In all three steps the authors impose Gaussian process (GP) priors to model the unknown functions, forming a set of deep GPs, one for each observed sequence. Additional structure is imposed on the function f by allowing it to model correlations between multiple outputs via a convolution with a number of independent processes. The key part of the paper is that the set of deep GPs is enforced to interact with each other via sharing information through the common middle layer, i.e., the function f is shared across the different Deep GPs. The model is learned via optimising the lower bound to the likelihood, which is obtained following a nested variational compression approach. Experimental results on toy and real world data demonstrate the advantages of the proposed model.  The paper is nicely written and I really enjoyed the fact that it targets a specific problem/application from the very beginning. It may seem a bit weird at first, however, the reader really knows what is expected to follow and can definitely make the connection between the modelling choices and the targeted challenges. I really liked the idea of sharing the middle layer and the inducing variables u_{f,d} across the multiple sequences. However, the authors should be a bit more careful with the used notation. From my understanding *boldfaced* f is basically the collection of the function f evaluated on the data a(x). If this is correct then when the authors refer to a shared *boldfaced* f throughout the manuscript can be very misleading, especially when we talk about sequences with different cardinality.  My biggest concern has to do with my understanding of the Psi statistics in Section 3.2 - First of all indices in this section are overloading the indices already defined. i is now used to denote the inducing point and n the point in the sequence, while i has been already used to denote the elements of X, i.e. X={x_i}_i=1:N. Can you please clean up this mess? - In both equations 6 and 7 we can see the Psi and Phi to be computed under some u. What is that u? Normally I would expect that the inducing input Z would be in the place of u. Have we assumed that the inducing inputs at the second layer are the inducing outputs of the previous layer? Then that would make sense, although, in that case we would need to integrate out also over u from the previous layer. In general I am very confused since in the same page in line 105 we have already defined different Z and u for f. Am I missing something? Please explain and correct me if I am wrong.  The evaluation procedure is targeted to highlight the advantages of the proposed model, although the captions accompanying the figures do not always fully explain what is happening: - For example in Figure 3 there is no description about the continuous lines that are draws from the model; the reader has to wait till the next experiment and specifically Figure 5 in order to acquire that information. - Furthermore, in Figure 4 there is no description whatsoever of what the middle row (purple) line demonstrates. Is this the recovered latent function? In general I do not understand how is it possible to associate points in that plot with points in the two turbines since the two sets have different cardinalities. Why this is not the case in the toy experiment in Figure 3?  Minor comments: - Figure 3 c & d, y labels for the plots in the third row should be g instead of y. Am I correct? - Same in figure 2 - References need update since you cite the arxiv version of many published papers.  ---After rebuttal--- The authors have addressed my comments. I enjoyed the paper. I believe it can get improved by including an additional experiment with more than two signals to align or a more general non-linear warping function on real data. Since I have not suggested that in my original review I trust the authors would include such an experiment in the final version, if the paper goes through. I am still positive about the submission.


Streamlining Variational Inference for Constraint Satisfaction Problems
Several algorithms for solving constraint satisfaction problems are based on survey propagation, a variational inference scheme used to obtain approximate marginal probability estimates for variable assignments. These marginals correspond to how frequently each variable is set to true among satisfying assignments, and are used to inform branching decisions during search; however, marginal estimates obtained via survey propagation are approximate and can be self-contradictory. We introduce a more general branching strategy based on streamlining constraints, which sidestep hard assignments to variables. We show that streamlined solvers consistently outperform decimation-based solvers on random k-SAT instances for several problem sizes, shrinking the gap between empirical performance and theoretical limits of satisfiability by 16.3% on average for k = 3, 4, 5, 6.

The article proposes a method for adding streamlining constraints in the survey propagation procedure, which improves its performance on random k-SAT problems. In line with classical survey propagation, they use a message passing procedure to compute approximate marginal. However instead of using the marginal to set single variables in the decimation process as previously done in survey propagation, the authors propose to add constraints based upon multiple variables. Iterating this procedure, more highly constrained problems are produced, which can then be solved by a downstream solver. The authors show that this idea improved upon a state-of-the-art survey propagation solver.  The article is well written, presents a novel and effective idea that I expect to see used in future SAT-competitions. While perhaps needed for the target audience of NIPS, I personally find the background section where previous algorithms are presented a bit too long. A substantial part of the paper is dedicated to the background that the reader could have easily been directed to by citations.   I have reviewed this paper at UAI 2017. I voted for acceptance at that time, and I am still leaning towards accepting this paper. One reviewer at UAI 2017 raised highly valid points, which is that incorrect and highly imprecise claims are made in several parts of the paper. I am very glad to see that the authors at least made some effort in revising these overclaiming statements. For example, now the authors only claim that their solvers outperform decimation-only solvers for random k-SAT instances in the abstract, instead of saying outperforming all solvers.  There are still a few places that I am going to bring authors’ attention. In particular, the authors should not make an implication that their method is analogous to streamlining random constraints in model counting, as in [2, 3, 14]. Agreed with reviewer 2 from UAI 2017, I think the contexts are entirely different, and the counting approaches in [2, 3, 14] simply cannot work with very short constraints, such as length-2 constraints used in this paper. I really think the authors should revise their statements in the third paragraph of the introduction.   Once this paper is converted to NIPS style, Figure 2 becomes way too small. Please enlarge the fonts in Figure 2. In addition, below are my detailed comments from UAI 2017.  On the technical side, there are a few directions that are not explored as much as would be ideal. First, the streamlining clauses are added based on the two most polarized variables. As an alternative, we can define a slightly different message passing algorithm which computes the marginal of the disjunction of all pairwise variables. Then we add streamlining constraints based on the pairwise marginal, instead of selecting the two most polarized variables assuming the independence. Will this be a better method?  The algorithmic design choices of section 3.3 could have been motivated more clearly. In particular, it would be interesting to plot how different choices affect the performance. Secondly, the constraints are just added to a single solver which is treated like a black box. To make the case of generality for the method, it would be preferable to use more solvers or have a metric of success that doesn’t depend on the selected black-box solver.   There are also a few presentation details that should be addressed. In section 3.3 the reader is directed to a theorem 1 which does not exist, I assume that it is supposed to be proposition 1. It is assumed that the three rows of figure 3 related to the same run - that could however be made clearer in the caption. In the introduction, only very recent work related to the phase transition of k-SAT is referred to. It would be good to cite several original publications.


Global Geometry of Multichannel Sparse Blind Deconvolution on the Sphere
Multichannel blind deconvolution is the problem of recovering an unknown signal f and multiple unknown channels x i from convolutional measurements y i = x i f (i = 1, 2, . . . , N ). We consider the case where the x i 's are sparse, and convolution with f is invertible. Our nonconvex optimization formulation solves for a filter h on the unit sphere that produces sparse output y i h. Under some technical assumptions, we show that all local minima of the objective function correspond to the inverse filter of f up to an inherent sign and shift ambiguity, and all saddle points have strictly negative curvatures. This geometric structure allows successful recovery of f and x i using a simple manifold gradient descent algorithm with random initialization. Our theoretical findings are complemented by numerical experiments, which demonstrate superior performance of the proposed approach over the previous methods.

This paper considered multichannel sparse blind deconvolution(MSBD) and proposed to solve a nonconvex optimization problem for recovering the underlying signal and filters. Under some technical assumptions, the authors showed that all local minima of the objective function correspond to the inverse filter of the ground truth signal up to an inherent sign and shift ambiguity, and all saddle points have strictly negative curvatures.  Finally, the authors proposed to solve this nonconvex optimization problem via a Riemannian gradient descend method.  My main comments are as follows.  1)  The authors did not clarify why they use the nonconvex optimization in (P1), what is the main idea behind this, and intuitively why this formulation will lead to correct recovery.  2) Compared to the reference [48], the difference is that this paper considered ‘multichannel’ (i.e., a bunch of channels (#N) to convolve with the ground truth signal and measure a bunch of (#N) observations). Despite this difference, these two papers share similarities: the objective is the same (except an additional summation appeared in this paper) and the main purpose is to analyze the geometry of the corresponding nonconvex optimization problem. However, it is unclear what are the technical difficulties and challenges for the multichannel case. It could be just a direct extension of the result in [48] and thus it is hard to assess the value of contribution and significance of this paper.   3) The Bernoulli-Rademacher model for the channels is too strong and is far away to real-world applications. The authors should clarify why using this assumption for the channels.    %%%%%% after rebuttal  %%%%%% I thank the authors for the response to my concerns. It is very good to see that all the results can be extended to Bernoulli-subGaussian model. The Bernoulli-subGaussian model would make the results more general and applicable. So I suggest to  at least add discussion about this in the final version. Also the explanation of the equivalence between maximizing \ell_4^4 and minimizing l0 norm could be in more detail. It is not very clear to see this relationship from the plot of \ell_4 norm.


Video-to-Video Synthesis
We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image translation problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without modeling temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generators and discriminators, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our method to future video prediction, outperforming several competing systems. Code, models, and more results are available at our website.

This paper focuses on video-2-video synthesis, i.e. given a real video the goal is to learn a model that outputs a new photorealistic and temporally consistent video with (ideally) the same data distribution, preserving the content and style of the source video. Existing image-2-image methods produce photorealistic images, but they do not account for the temporal dimension, resulting in high-frequency artifacts across time. This work builds on existing image-2-image works and mainly extends them into the temporal dimension to ensure temporal coherence. By employing conditional GANs the method provides high-level control over the output, e.g. certain parts of the dynamic scene can be introduced or replaced though segmentation masks. Although the theoretical background and components are employed from past work, there is significant amount of effort in putting them together and adding the temporal extension. Although this work does not solve the problem per se, it seems like an important step for which the community would be interested in, while the authors also explicitly promise to release their implementation. The results look reasonable, though I would like to see a more involved discussion about current disadvantages and future directions. Furthermore, some clarifications are needed and suggestions for further experiments are given. Although I am not an expert in this narrow field, I am leaning towards supporting this work, on the condition that feedback will be incorporated. More detailed feedback follows below.   - L10, L38 - I am not very convinced about the novelty of the 'new generator and discriminator architectures', in the technical section they are discussed as straightforward extensions.  - L167-168 - My understanding is that this is not what you do. So I am not sure about the use of this comment in the manuscript. Can you please comment further? What is used for the provided results in the manuscript and supplementary material? The suggested speedup trick would force hallucination for the foreground areas, which is in contrast with the goal of this work as explicitly mentioned in L3 (and elsewhere), temporal consistency and content preservation would be harder for the foreground. If, in contrast to my understanding, this is what you do, can you please be more specific about the trade-offs?  - L119 - The phrase 'between 0 and 1' sounds a bit ambiguous, the values are binary.  - Eq4 - Maybe you can help the reader with more intuition, e.g. the first part corresponds to non-occluded regions (synthesising by warping) while the second corresponds to occluded regions (synthesising by hallucinating).  - Eq7 - In the first term, shouldn't the superscript for w be i-2 ? - Eq8, L153 - It is counter-intuitive to see the two lambdas to have the same value. Maybe you can combine them in one lambda and move it from Eq8 to Eq5 ? - The lambda(s) in Eq8(Eq5) is an important steering weight for the training objective, how do you set this? I guess it is too expensive to provide evaluation, so some intuition would be helpful. - L159 - Please compare with L160, it seems there is a \widetilde{x} (etc) missing in H_F. - L192, L193 - I would love to hear some more specific details in the supplementary material, for better self-consistency. Why is this mentioned here and not after Eq5? - Nice idea to include videos in the pdf. There is a clear improvement in the qualitative results, however I would like to hear some discussion about current disadvantages and pointers for future directions. In the long videos of the supplementary material replacing objects through the segmentation masks is interesting but has obvious artifacts (buildings, trees), any pointers for improvements?  - L108 - Can you please discuss how you set L=2? It would be great if you could provide at least qualitative comparison for a short video that gives intuition to the reader. Furthermore, could you please add a comment on what is the price to pay for a bigger L in terms of modelling/training difficulty? - The method employs optical flow estimation and occlusion mask estimation (L123), and both are sources of noise. It would be great if you could replace estimation of them with ground truth on some synthetic dataset, this could give pointers and prioritisation on what would improve your results (e.g. parts of lanes disappear/blur after some frames).   - minor typos:    - L120 - 'of the image'    - L274 - 'allows the user'    - L275 - 'synthesise a new' - Thank you for explicitly promising to release code/models, reproducibility could be hard without this.   ==================================    Updates after authors' response:  - I welcome the authors' responce and clarifications. My new comments are again towards constructive refining of the manuscript. Some additions are asked below, in order to incorporate the responce in the manuscript in case of acceptance. I understand that there are space limitations, so the authors should do their best to communicate the points in the main paper even more briefly than in the rebuttal, but worst case they should be in the supplementary material and video.  - "Using ground truth flows" - Thank you for performing the requested experiment. Please incorporate in the paper and in the supplementary video. - "Limitations and future work" - Thank you for the discussion. I often feel that in certain communities this is underplayed, while it points the community to interesting steps. Again, please incorporate in the paper and in the supplementary video. For point 3 I am not fully convinced about the coarser semantic labels (training less affected by the label shapes), but maybe I am missing something. - Thank you for offering to downplay the "new" term and novelty of architecture. I do agree that the extension is not trivial. I am not sure about the comment on combining different resolution, if I remember correctly this is not tackled.  - "Synthesis in the foreground area" - Please add in the paper the comment about flickering and "visual quality". I am not sure what you mean with the term "visual quality", so please reword to communicate it better. The flag in the code is a great idea. Please add a qualitative comparison in the video, it would be very informative wrt the trade-off between image quality and decision on FG synthesis.  - "Setting parameters and losses" - Please add very brief comment on similarity for 1..10.  - L=2 is chosen but L=4 (more temporal info/frames) gives worse performance. This shows that the temporal extension is not trivial, and points that more future work is needed to cleverly use temporal information. This should be part of "future work" and must be mentioned. This is not to downplay this work, just to underline that the problem is non-trivial. Please mind that the resolution due to time constraints was decreased for this experiment.


How Does Batch Normalization Help Optimization?
Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training.

Summary  The authors present a new justification of the effectiveness of Batch Normalization. They show empirically that what was previously believed to be the main advantage of Batch Normalization, i.e. reducing Internal Covariate Shifts, is actually not something that is modified by this reparametrization and may even be amplified by it. In light of this, they present a new analysis using lipschitz smoothness and how it stabilises both loss landspace and  gradients leading to a better (or easier) optimization.  This work is very much needed for a better understanding of the underlying effects of batch normalization on optimization and generatlization and it is a natural extension of the recent literature of theoretical deep learning.  Quality:  The empirical proof on the internal covariate shift is too qualitative. The  Figure 1 gives at best a rough idea of what is going on but there is no quantititave measure of the rate of change of the moments of the distribution for instance. Subsection 2.2 is a good attempt however at quantifying the  shift.   The analysis of section 3 falls somehow short. A strong relation between the  lipschitz smoothness and optimization should be explicitly given in the paper.  The empricial analysis is not broad enough in my opinion. As listed in this paper, there is a plethora of alternatives to batch-normalization that have been proposed in the past years. Since they all have their up and downs, comparing them on the metrics of ICS as proposed in this paper or lipschitz smoothness would most probably be enligthning.  Clarity:  The paper is well written and concise. The storyline is clear and easy to follow.   Originality:  As far as I know this is the first analysis to shed light on the stability and smoothness effect of batch normalization as its most important feature for helping optimization.   Significance:  This paper is important for two reasons. 1) Batch Normalization is a widely used technique, yet its important underlying properties are not very well understood. 2) Since Batch Normalization seems to help with both optimization and generatlization, a better understanding of it could give fundamental insights for the subfield of theoretical deep learning.


Exponentiated Strongly Rayleigh Distributions
Strongly Rayleigh (SR) measures are discrete probability distributions over the subsets of a ground set. They enjoy strong negative dependence properties, as a result of which they assign higher probability to subsets of diverse elements. We introduce in this paper Exponentiated Strongly Rayleigh (ESR) measures, which sharpen (or smoothen) the negative dependence property of SR measures via a single parameter (the exponent) that can be intuitively understood as an inverse temperature. We develop efficient MCMC procedures for approximate sampling from ESRs, and obtain explicit mixing time bounds for two concrete instances: exponentiated versions of Determinantal Point Processes and Dual Volume Sampling. We illustrate some of the potential of ESRs, by applying them to a few machine learning problems; empirical results confirm that beyond their theoretical appeal, ESR-based models hold significant promise for these tasks.

### Post-Rebuttal ###  I have read the rebuttal and would like to thank you for the clarifications.  ##################  # Summary The paper introduces a concept of exponentiated strongly Rayleigh (ESR) measure motivated by the trade-offs between diversity and quality of items in a selected subset of a ground set with n items. A measure \mu over the subsets of a ground set of n items is called strongly Rayleigh (SR) if its generating polynomial  P_{\mu}(z_1, ..., z_n) = \sum_S \mu(S) \prod_{i \in S} z_i is `real stable'. The authors take the definition of a `real stable' polynomial from [10] and define it as a polynomial without complex roots. An exponentiated strongly Rayleigh measure \mu is defined as a measure such that there exists a strongly Rayleigh measure \nu satisfying \mu(S) \prop \nu(S)^p for some p >= 0. Now, for p >1 the ESR measure promotes diversity whereas for p < 1 it allows less diverse items at the expense of their quality. For p=0 the ESR measure is uniform and for p=\infty it concentrates at the mode.  # Comments/Details - The first theoretical result is Proposition 1 which states that there exists an ESR measure which is not a SR measure. This is a key result for establishing the `novelty' of ESR measures, showing that they are not contained in the class of SR measures. The proof of this proposition might need a clarification. In particular, if in the proof of this proposition one sets z=-l_{22}^p + u and w=-l_{11}^p + \overline{u} with u \in C such that || u ||^2 = l_{11}l_{22} - det(L^p) then one obtains a complex root of the polynomial p(z, w). In my understanding this then does not align with Definition 1. Moreover, the proof of Proposition 1 contains a number of claims without a proper mathematical derivation. If these results are from other papers, then those should be cited accordingly. Otherwise, a complete proof of the proposition is required.  - The second theoretical result is Theorem 1, where it is shown that there exists a choice of p and a non-trivial matrix for which its exponent defines a determinantal point process that is an SR measure. This result is novel and establishes that some ESR measures remain SR.  - To sample from an ESR measure the authors consider two algorithms. The first algorithm is an independent Metropolis--Hastings sampler with a proposal distribution given by a SR measure. It is quite strange that the authors do not cite the paper by Metropolis or some survey on MCMC methods because there is nothing novel about this algorithm. The second algorithm is pretty much the one published in [29]. I do not see any significant novelty in this part of the paper.  - In order to study the mixing properties of the chains for sampling ESR measures the authors introduce a concept of closeness of measures (Definition 3) and provide a bound for the parameter defining it in Proposition 3. Following this, the main theoretical results are given which provide the mixing bounds for chains defined by Algorithms 1 and 2. Parts of these proofs can be obtained by mimicking previous derivations such as [5].  - In the concluding part of the section on ESR measures the authors focus on determinantal point processes and ESR measures defined by exponentiating a positive definite matrix defining such a process. For this specific case, the authors provide bounds on the parameter defining the closeness of distributions in mixing bounds for the considered sampling algorithms (see the comment on Definition 3).  - The approach is evaluated empirically via three experiments: i) an experiment illustrating the mixing properties of the chain, ii) anomaly detection problem, and iii) Nystroem method with ESR landmarks (this experiment is replicated from the paper on DPP-Nystroem from [30]). The experiments illustrate the utility of the exponent in trade-offs between quality and diversity among items from a selected subset.  ### Overall the main part of the paper is well written and easy to follow. In the appendix, it is quite difficult to follow the proof of Proposition 1 because of the claims that are expected to be accepted without a proof. In the summary of the contributions, it might be beneficial to be more explicit and state that the proposed algorithms are not novel contributions of the paper. The theoretical results providing mixing bounds for these samplers in combination with ESR measures are novel and interesting. Still the impression is that these are just minor adaptations of the previous work. I do, however, appreciate the flexibility introduced via the exponent p in ESR measures. It would be nice if the authors provided an ESR measure that is not in the context of determinantal point processes. The focus on determinantal point processes precludes other contributions and the utility of general ESR measures. ###


Incorporating Context into Language Encoding Models for fMRI
Language encoding models help explain language processing in the human brain by learning functions that predict brain responses from the language stimuli that elicited them. Current word embedding-based approaches treat each stimulus word independently and thus ignore the influence of context on language understanding. In this work, we instead build encoding models using rich contextual representations derived from an LSTM language model. Our models show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area. By varying the amount of context used in the models and providing the models with distorted context, we show that this improvement is due to a combination of better word embeddings learned by the LSTM language model and contextual information. We are also able to use our models to map context sensitivity across the cortex. These results suggest that LSTM language models learn high-level representations that are related to representations in the human brain.

The authors attempt to study how variable context length relates to the fMRI recordings of subjects who are listening to a story. They investigate variable context lengths through a 3-layer LSTM language model and test how well they can predict brain activity using representations from different layers and different context lengths.  Quality: I completely agree that context is important in representing language in the brain. However, I believe this paper has not done enough to address very likely confounders.   First off, it’s not clear how the encoding model is trained on the actual data set. As far as I understand, Figure 1 shows the pipeline at test time, not train time. How are the beta weights used for the fMRI data prediction in Section 5.1 obtained? If they are trained using the fMRI recordings and the corresponding stimuli from the data described in Section 2, then how did you handle multiple repetitions of a word in different contexts? I believe this is extremely important and can explain some of the results.  The corruption of context results in Section 5.2 are expected because the LSTM is trained to model language, not random “nonsensical” sequences of text, so of course its representations will deteriorate. When the representations are bad, the prediction of fMRI data is also bad, so the encoding model performance decreases. A much more convincing control would be to look at the same word in two different contexts m1 and m2 (of same length), and compare the performance of the encoding model of predicting the brain activity of the word in one context from the LSTM representations of both m1 and m2. Then you can also vary the length of m1 and m2.   The map results are speculative and are only for 1 subject -- why not show a mean over all subjects, or at least include all of the subject maps in the supplementary if these results are consistent?   Clarity: It is not clear how the encoding model was trained, which is extremely important. In general, it would be good to see a lot more detail about the experiments than is offered. Also it is unclear what kind of correlation is used for the encoding model.  Novelty: There does not seem to be methodological novelty. On the neuroscience side, using language models to show that context is important in the brain is also not a novel idea -- the paper even cites (though in an off-hand manner on line 41) one paper that uses an RNN LM to show that one can more accurately predict the brain activity for the next word by using the previous context (hidden layer of the RNN) than by using the embedding of the current word. The idea to study variable length context is interesting and novel, but I don’t believe that the experiments appropriately address this question.  Significance: If my concerns are not correct and the brain map results are consistent across subjects, then this work is significant for the neuroscience community.  Post rebuttal: The authors addressed two of my main concerns in their rebuttal: they showed results across additional subjects and presented results from a context swapping control experiment. However, in reading through the rebuttal and the paper an additional time, I have realized that there may be an additional confounder in the results, that is not controlled for in any of the experiments and could explain the main result.  It is known that the LSTM word representations contain contextual information, and that these representations are sensitive to the nearby context length and word-order within this nearby context (https://arxiv.org/pdf/1805.04623.pdf). So the question is whether the improvement in encoding model performance with longer context is due to the increase in this contextual information in the LSTM representations (as the authors hypothesize), or to something else (a confounder)?  One possible confounder is that increasing the context length of word W makes it more likely that word W will appear in this context, thereby allowing the LSTM representation of W in the longer context to be more influenced by W than it was in the shorter context. In this case, an encoding model trained to predict brain activity elicited by W from the longer context LSTM representations does not necessarily perform better due to the additional context, but possibly due to the fact that the LSTM representation is more similar to W. The more frequent W is, the worse the effect of this confounder could be. Considering that the presented results are over all words in the test set, including very frequent stop words, this alternative explanation could account for much of the improvement of the encoding model when the context length is increased.  Control experiment: The randomized context experiment they perform in the original paper does not control for the number of repetitions of word W, and it also has another downside that I pointed out in my original review. The swapped context experiment, as performed by the authors, also does not control for this confounder, because the swapped context is chosen so that it “has the fewest words in common with the original text” [line 30 in rebuttal]. An experiment that would account for the presented confounder is similar to the swapped context experiment, but would have the same number of occurrences of word W in the original and the swapped context.   Quantification of result significance: it is good that the authors present cortical map results for 3 additional subjects, however they are difficult to compare visually for consistency with the claimed results. In addition, it is difficult to judge how significant an increase in “encoding model performance” is with increased context without any notion of chance or hypothesis testing.   For the above reasons, my score remains the same and I recommend the paper for rejection. I strongly disagree with the other two reviewers.


Verifiable Reinforcement Learning via Policy Extraction
While deep reinforcement learning has successfully solved many challenging control tasks, its real-world applicability has been limited by the inability to ensure the safety of learned policies. We propose an approach to verifiable reinforcement learning by training decision tree policies, which can represent complex policies (since they are nonparametric), yet can be efficiently verified using existing techniques (since they are highly structured). The challenge is that decision tree policies are difficult to train. We propose VIPER, an algorithm that combines ideas from model compression and imitation learning to learn decision tree policies guided by a DNN policy (called the oracle) and its Q-function, and show that it substantially outperforms two baselines. We use VIPER to (i) learn a provably robust decision tree policy for a variant of Atari Pong with a symbolic state space, (ii) learn a decision tree policy for a toy game based on Pong that provably never loses, and (iii) learn a provably stable decision tree policy for cart-pole. In each case, the decision tree policy achieves performance equal to that of the original DNN policy.

# Paper ID 1246  Verifiable RL via Policy Extraction  ## Summary  The main idea of the paper is to use a strong learned policy as an expert (e.g., deep neural network) in an imitation learning setup. A loss function designed to minimize the gap between the best and worst action (as defined by the expert) is used to train a CART policy. Three verifiable properties (correctness, robustness, stability) can be associated with the tree from the piecewise linearity of the partitions imposed by the tree policy that the original expert policy (e.g., a deep neural network) may not possess. However, correctness and stability require the system dynamics to be known. The empirical evaluation is somewhat limited. For example, experiments are shown on two relatively small domains (Atari Pong, cart-pole) where the model dynamics are simple enough to specify or approximate.  Overall, the paper is clearly well written. The algorithmic ideas are intuitively clear and seem sound to me. However, I'm a bit unclear about the feasibility / applicability of the verification process in other domains. The experimental section is also a bit limited. Overall, there are some interesting ideas in the paper but it's a bit difficult for me to evaluate due to the limited discussion and experiments. Detailed comments follow.   ## Detailed Comments  - The paper introduces a vanilla imitation learning setup for the task of extracting a policy from a given oracle or expert.  - A loss function designed to minimize the worst-case gap across the expected state distributions is defined. A decision tree learner is used on a dataset resampled under the new loss function which has the effect of "compiling" the original expert into a decision tree trained to minimize the expected worst-case loss across the distribution of states induced by the learned tree policy. This learning algorithm (Q-DAGGER) is the key algorithmic contribution.  - Intuitively, the main idea is to focus the training on states where the gap in the values between the best action and the worst action is largest. These are the "critical states". Standard 0-1 loss doesn't treat these states differently from other states. The idea here is that the learner should focus on avoiding the worst-case loss in states (i.e., $l_max$ is small). If such a policy can be found, then the bound improves over vanilla DAGGER.  - It's not clear to me how problems with "critical states" are distributed. For example, are the common control tasks in MuJoCo / OpenGym good candidate problems for this approach? Additional discussion characterizing problems where the methods are most / least applicable would be nice to have.  - Assuming such a problem is given, the approach proceeds by modifying the loss function to incorporate the worst-case gap in values. This requires an expert that can provide a worst-case "gap" (e.g., Q values, expert policy probabilities) between the best and worst action in any state. This seems reasonable to me.  - A tree policy can now be learned via CART in an iterative fashion (i.e. DAGGER). The main idea is to reweight the accumulated examples in the dataset using the worst-case gap values described above. This is the policy learned by the VIPER algorithm. This part is also very straightforward and well described.  - Next, the paper proposes three properties that are verifiable by VIPER under appropriate conditions. Correctness and stability require the model dynamics to be known while robustness does not. Here, I do think there needs to be additional discussion on the feasibility of obtaining the dynamics for other problems. Also, even if the dynamics are known, the difficulty of specifying the constraints in the form required by VIPER is unclear to me. A discussion of the complexity of these properties on MuJoCo tasks, for example, may be good to have. Discussion of these in Section 3 would strengthen the paper significantly, in my opinion.  - The empirical evaluation is performed on the Pong domain and cart-pole. It seems like verifying correctness and stability is rather difficult for a given domain. It might be worth discussing. Next, I'd be curious to know why robustness is only evaluated for Atari Pong and not cart-pole? Perhaps the paper could focus on a single property (e.g., robustness) and show experiments on additional domains. In its current form, the experimental section feels a bit limited. Additional experiments and their analysis would be very nice to have.  - Overall, it seems like the choice of loss function is orthogonal to the verification requirements, which only require the decision tree. Is my understanding correct? If yes, then other loss functions and their impact on tree size and properties would be quite interesting to study. For example, I'm curious to understand why exactly the trees under the 0-1 loss are larger for a particular reward target. How much of the tree size depends on the worst-case loss function seems like something worth exploring further.  Update -------- I thank the authors for their response. My overall score remains unchanged.


Sparsified SGD with Memory
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. Various recent works proposed to use quantization or sparsification techniques to reduce the amount of data that needs to be communicated, for instance by only sending the most significant entries of the stochastic gradient (top-k sparsification). Whilst such schemes showed very promising performance in practice, they have eluded theoretical analysis so far. In this work we analyze Stochastic Gradient Descent (SGD) with k-sparsification or compression (for instance top-k or random-k) and show that this scheme converges at the same rate as vanilla SGD when equipped with error compensation (keeping track of accumulated errors in memory). That is, communication can be reduced by a factor of the dimension of the problem (sometimes even more) whilst still converging at the same rate. We present numerical experiments to illustrate the theoretical findings and the good scalability for distributed applications.
compression can hurt the performance, unless it is implemented in a clever way: 1Bit-SGD [33,37] combines gradient quantization with an error compensation technique, which is a memory or feedback mechanism. We in this work leverage this key mechanism but apply it within the more general setting of SGD. We will now sketch how the algorithm uses feedback to correct for errors accumulated in previous iterations. Roughly speaking, the method keeps track of a memory vector m which contains the sum of the information that has been suppressed thus far, i.e. m t+1 := m t + g t − comp(g t ), and injects this information back in the next iteration, by transmitting comp(m t+1 + g t+1 ) instead of only comp(g t+1 ). Note that updates of this kind are not unbiased (even if comp(g t+1 ) would be) and there is also no control over the delay after which the single coordinates are applied. These are some (technical) reasons why there exists no theoretical analysis of this scheme up to now.
In this paper we give a concise convergence rate analysis for SGD with memory and k-compression operators 2 , such as (but not limited to) top-k sparsification. Our analysis also supports ultra-sparsification operators for which k < 1, i.e. where less than one coordinate of the stochastic gradient is applied on average in (1). We not only provide the first convergence result of this method, but the result also shows that the method converges at the same rate as vanilla SGD.

This work analyzes a variant of SGD designed to incur lower communication costs in distributed systems. This variant of SGD combines sparsification with memory. At each iteration, the algorithm sparsifies the stochastic gradient by sending only the k (out of d) largest coordinates (in absolute value) or a random k coordinates to other nodes in the system. The portion that was not transmitted is added to the next stochastic gradient, and their sum is sparsified in the same manner. The authors show that without memory, the convergence of this method could be adversely affected, but that by maintaining this memory, the convergence rate resembles that of SGD, modulo a d/k factor. The main theory of the paper uses perturbed iterate analysis to analyze sparsified SGD with memory. The authors also give detailed experiments showing that selecting the top k coordinates outperforms random k coordinates, and comparing the method favorably to QSGD.  Quality: The submission seems to be theoretically sound. Many of the techniques are straightforward adapatations of techniques from perturbed iterate analysis. The primary theoretical novelty is Lemma 3.2 which uses the fact that sparsification can be seen as a kind of contraction operator (Definition 2.1) and analytic estimates to analyze the norm of the memory vector over time. Moreover, the experimental results show (as prior works have shown) the efficacy of the strategy.  I have only two main comments about this part. The first is that some discussion of how this analysis might carry over to the convex or even non-convex setting would be illuminating. Seeing where and how the theory breaks could help spur the community to fix these issues. The second has to do with Figure 3. The authors correctly note that sparsified SGD with memory can be much more sparse than QSGD. The authors try 2, 4, and 8 bit QSGD (as the authors of the original QSGD paper do), but they only display 4 and 8 bit QSGD for one experiment and 2 and 4 for the other. It is not immediately clear why this omission occurs.  Clarity: The paper is well-written throughout. The underlying problem is well-motivated in the introduction, and a very useful technical motivation is given in Section 2.2. Comparing the variance blow-up reduction to that of mini-batch SGD was particularly insightful. The theory is organized well, and the experiments setups are detailed.  A few comments that might help the readability throughout. First, more clearly differentiating between sparsification and quantization may be helpful. For instance, in line 29 you refer to a quantization function as sparse(), even though quantizations need not be sparse. Second, in the appendix every equation is numbered. It might be more helpful to the reader to only number the important ones. Intermediate steps need not be numbered.  Originality: The theoretical results in the work are to the best of my knowledge novel. They are very similar in scope to results concerning the use of 1-bit SGD with memory, but apply to sparsification instead. The authors do a good job of citing other work that has analyzed SGD with memory empirically and differentiate themselves by presenting their theoretical work.  While the authors do a good job of citing all the related work, some of the related work section could be fleshed out or improved. For instance, lines 51-52 state that "Sparsification methods reduce the number of non-zero entries in the stochastic gradient," which while true, ignores all nuance in different methods. Short general explanations of what some of the methods do would be useful, especially for QSGD which is later compared directly to SGD with memory. Last, some of the phrasing in the related work section is based on opinion or conjecture instead of fact. For instance, line 55 says that sparsification methods "can only work" when memory is incorporated. While I agree that vanilla sparsification methods can use memory to their benefit, to suggest they are never effective without them or that there are no other possible methods is not necessarily correct.  Significance: The theory is useful for confirming what empirical studies have seen in practice. The theory is also significant for its simplicity and conciseness. In terms of scope, it is similar to other results (such as 1-bit SGD with memory), and only extends to strongly convex functions. Nevertheless, its mathematical elegance is worth highlighting. Its experimental results comparing SGD with memory to QSGD are somewhat significant as well, though due to the number of variants of QSGD and parameters to be set, a more comprehensive analysis might be warranted.  All in all, I believe that the paper deserves acceptance to NIPS. It is well-written, mathematically concise, and gives a step forward in understanding how the use of memory can help sparsification/quantiztion techniques.


The Effect of Network Width on the Performance of Large-batch Training
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however it besets the convergence of the algorithm and the generalization performance. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that-for a fixed number of parameters-wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.

It has been wide discussed on how to develop algorithms allow large batches, so that one could train neural networks in a distributed environment. The paper investigates the effect of network width on the performance of large-batch training both theoretically and experimentally. The authors claim that with the same number of parameters, it is more likely to train neural networks using proper large batches easily with a wide network architecture. The theoretical support on 2-layers linear/nonlinear networks and multilayer linear networks is also given.   The paper is well-written and easy to follow. It extends the analysis of Dong el.s' paper about gradient diversity and addresses partly the open question proposed in their works about neural networks. The experiments are well-designed to show the advantage of wide network in many aspects. Plenty of extensive demonstrations (multilayer non-linear or residual networks) are provided and the results are informative. I would therefore vote an acceptance for the paper. However, I have some concerns and I hope it could be addressed   1. There are plenty of unused hyper-parameters in Theorems (c_1, c_2, c_max, c_sup, W^*, W, etc.) and also it's vague to say "with arbitrary constant probability", "sufficient large" or "high probability" in formal stated theorems. If you do want these constants, could be better to have some dependence on it. 2. In the proofs of Theorem 1, probability \delta is used to derive the final bound of B_S(w), while I'm not sure, how could you derive the last equation over line 443? should \delta be bounded? please also show how you handle the negative \Theta item in the nominator.  3. The conclusion of Theorem 2 and 3 shows the ratio of the expectation, and it does not imply a direct bound for the batch threshold B_S(w), so please explain more about the intuition that supports your conjecturing, otherwise the theoretical conclusion might be over-claimed. 4. To measure the convergence performance among different networks architectures and batch-size, a fixed convergence threshold is pre-defined, e.g., MNIST to 96% accuracy. I'm wondering is the value sensitive to your claims? It usually takes more epochs to achieve better accuracy in the final phase of training, considering the probably better expression of deep nets, what would happen if you trying to train MNIST to 100% accuracy? 5. I maybe misunderstanding somewhere, but in figure 3, how could you have same parameters with say (b) K=21, L=1 and K=17, L=10?  Based on the comments above, I would therefore vote a weak accept to the work, but I would like to appreciate it if you could make it a better work.  minors:  1. Line 33, duplicated reference [18]; 2. Line 256, redundant word "this" 3. Missing clarification of order symbols (\Theta and \Omega) in Theorems;


Video Prediction via Selective Sampling
Most adversarial learning based video prediction methods suffer from image blur, since the commonly used adversarial and regression loss pair work rather in a competitive way than collaboration, yielding compromised blur effect. In the meantime, as often relying on a single-pass architecture, the predictor is inadequate to explicitly capture the forthcoming uncertainty. Our work involves two key insights: (1) Video prediction can be approached as a stochastic process: we sample a collection of proposals conforming to possible frame distribution at following time stamp, and one can select the final prediction from it. (2) De-coupling combined loss functions into dedicatedly designed sub-networks encourages them to work in a collaborative way. Combining above two insights we propose a two-stage framework called VPSS (Video Prediction via Selective Sampling). Specifically a Sampling module produces a collection of high quality proposals, facilitated by a multiple choice adversarial learning scheme, yielding diverse frame proposal set. Subsequently a Selection module selects high possibility candidates from proposals and combines them to produce final prediction. Extensive experiments on diverse challenging datasets demonstrate the effectiveness of proposed video prediction approach, i.e., yielding more diverse proposals and accurate prediction results.

The authors propose a method for video prediction with a particular focus on the problem of minimizing blur in predicted frames while optimizing for precision in motion forecasting. The main contribution is an introduced sampling module that explicitly separates different modes in the motion component and thus ensures disentangling the uncertainty in possible futures from visual quality of predicted frames.   It is an interesting and to the best of my knowledge original work, both in terms of the theoretical framework and the obtained results. The authors test their implementation on three commonly used benchmarks (MovingMNIST, RobotPush and Human3.6M) by predicting ~10 frames in the future and show significant visual improvements compared to the state-of-the-art, both in terms of visual quality and plausibility of movements.  Overall, it was an enjoyable read - however, the clarity of the manuscript can be improved, as at this point some parts are rather difficult to parse. It also contains a certain number of typos and minor grammar mistakes (some of them are listed below).  In general, the necessity of each of the proposed blocks could be more explicitly explained and justified by ablation studies. The "combiner" module, for example, should be better motivated: the principled difference between the top candidates and the final result is unclear. What is the meaning of masks in this case and how do they look in practice?  It would also be interesting to analyze the performance, possible failure modes and the nature of learned conditional distributions of the selector module - for now there is no discussion on this in the paper.  I'm not totally convinced that the experiment described in Table 1 is necessary - these results are rather intuitive and expected. It could make sense to sacrifice it in the interest of space and give more detailed description of the main contributions.  The experimental studies are relevant but not exhaustive. It looks like the authors do not provide any quantitative evaluation and comparison with the state-of-the-art on the mentioned Human3.6M dataset. For RobotPush, the only quantitative comparison is expressed in inception scores.  Figures 1-3 would be more helpful if the authors added notations used in the text.  There is something funny happening with notations for time stamps. L93-94 state the problem as "given x_t, predict x_{t+1}". However, eq. (1) contains x_t and x_{t-1} instead. The next paragraphs (L106-113) talk about all three of them: x_{t-1}, x_t and x_{t+1}. Eq. (5) should probably read k=min|x_{t+1}-\hat{x}_{t+1}| (otherwise, it points to a degenerate solution \hat{x}_t=x_t).  Assuming that in reality two frames x_{t-1} and x_t are given to predict the third frame x_t, one wouldn't expect the network to generalize beyond linear motion - is it indeed the case? In practice, does extending the temporal dimension of the input affect the accuracy of motion estimation in longer term?  The relationship between W, \Delta W and proposals X is not clear to me. Figure 2 hints that W are convolutional kernels of the sampling decoder (but it's a 3-layer ResNet, according to Section 3.3) but doesn't show how Ws are generated. There is an arrow saying "Element-wise addition" but is has a single input \Delta W_i. If active components of W are different for different proposals \hat{x}_t^i, what prevents us from learning them directly using eq. (5) by selectively updating sub-bands W_i?  Symbol W is overloaded and stands for both frame width and kernels.  The discriminator in Eq. (1) optimizes for "realism" of proposals x_t regardless of their "compatibility" with the past frames x_{t-1}. I realize this compatibility aspect is taken into account later by predicting "confidence scores", but I'm curious how conditional and unconditional discriminators would compare in this setting (if the authors have experimented with this).  L144: why exactly is the per-pixel L1 norm better applicable to linear motion?  Section 3.3. The authors mention curriculum learning "along the temporal dimension" but do not explain how it is implemented in their framework.   Table 2, third column: it's not clear what is meant by "inputs" and "outputs" (are the inputs processed / outputs predicted at once or in a sliding window manner? is it the same for all frameworks? how is it compatible with the description in Section 3?)  How do the authors explain that the performance on Human3.6M simultaneously goes down in terms of "reality" (23.1%) but improves in terms of "image quality" (40.8%)? (Table3, last column).  Section 4, Figure 4. Could the authors comment on why the proposed method is compared to DFN and SVG on Moving MNIST (in terms of PSNR and SSIM), but to SV2P and CDNA on RobotPush (in terms of inception scores)? It would be also interesting how performance of these methods compares to trivial baselines (copying the last observed frame, warping with optical flow).  L262-267. Which dataset do the author refer to? Please also mention it in the caption of Table 4. The performance is improved by increasing K from 1 to 2 - what happens when N and/or K grows further? Again, I would like to see more empirical evidence justifying the presence of each component: can we remove the selector (i.e. set K=N)? or the combiner?  Nitpicking: L30: is is possible -> it is possible L36: As show -> As shown L48: Nitish et al. -> Srivastava et al. L49: [32] propose -> [32] proposes L54: methods suffers -> method suffer L61: inference is not a verb L76: 34 twice L117: Rivera et al. -> Guzman-Rivera et al. L228: As show -> as shown Ref [23]: CoRR -> ICLR, 2014 Ref [29]: CoRR -> ICLR, 2016 etc.


Adaptive Online Learning in Dynamic Environments
In this paper, we study online convex optimization in dynamic environments, and aim to bound the dynamic regret with respect to any sequence of comparators. Existing work have shown that online gradient descent enjoys an O(
where T is the number of iterations and P T is the path-length of the comparator sequence. However, this result is unsatisfactory, as there exists a large gap from the Ω( T (1 + P T )) lower bound established in our paper. To address this limitation, we develop a novel online method, namely adaptive learning for dynamic environment (Ader), which achieves an optimal O( T (1 + P T )) dynamic regret. The basic idea is to maintain a set of experts, each attaining an optimal dynamic regret for a specific path-length, and combines them with an expert-tracking algorithm. Furthermore, we propose an improved Ader based on the surrogate loss, and in this way the number of gradient evaluations per round is reduced from O(log T ) to 1. Finally, we extend Ader to the setting that a sequence of dynamical models is available to characterize the comparators.

This paper studies online convex optimization in dynamic environments, where one wishes to control the regret with respect to sequences of comparators, as opposed to a static comparator. The complexity of a comparator sequence is measured by its path length P_T, and the goal is to obtain an optimal regret bound simultaneously for all path lengths.  A first bound in this setting was established in the pioneering paper [1], which showed that online gradient descent (OGD, projected on the convex compact domain) with step-size 1/sqrt(T) achieves an at most T^{1/2} (1 + P_T) regret for all comparator sequences. However, there is a gap between this upper bound and the (T (1+P_T) )^{1/2} lower bound on worst-case regret established in Theorem 2 of the present paper, which is the first lower bound for this problem. On the other hand, if the path length P_T one wishes to compare against is known in advance, optimally tuning the step-size of OGD with respect to P_T in the OGD regret bound yields optimal (T (1+P_T) )^{1/2} regret for this path length.  Building on this observation, two algorithms are introduced which satisfy (T (1+P_T))^{1/2} regret simultaneously for all path lengths P_T, matching the above lower bound, up to a doubly logarithmic term in T. The first algorithm, called Ader, consists of a master which aggregates O(log T) instances of OGD with step-sizes in a geometric grid, using an expert algorithm (namely exponential weights). The standard expert regret bound implies a regret bound with respect to all step-sizes, and hence adaptivity to P_T (Theorem 3). As mentioned in the paper, the idea of aggregating different instances of OGD on a grid of step-sizes was considered in [2], to obtain different kinds of adaptivity guarantees. Note that Ader requires to query O(log T) values and gradients at each step, while OGD only needs 1 gradient. To address this, the authors introduce a variant of their algorithm, called "improved Ader", which consists of Ader run on linearized surrogate losses based on the gradient at the point chosen by the master algorithm (hence, only this gradient is used). Using the standard "gradient trick" (regret with respect to the original loss functions is bounded by regret with respect to the linearized losses) and the regret bound for Ader, a similar optimal (T (1+P_T))^{1/2} regret bound is obtained for improved Ader (Theorem 3). Finally, the algorithm and results are adapted to the setting when one measures the complexity of the sequence in terms of deviation with respect to a dynamical model rather than path length, introduced in [3].  This paper is superbly written and easy to follow; in particular, the "Related Work" section provides a clear and detailed account of existing work on the subject. The lower bound and adaptive upper bound for this problem are novel. The approach is natural and the techniques used somewhat straightforward, which I would argue is a plus rather than a minus.  Note: The upper and lower bounds are almost matching, up to a log log T factor. While this additional factor can be safely ignored as almost constant, it turns out that it can be avoided altogether, as follows. Instead of putting a uniform prior on the exponential grid of step-sizes, set a mass 1/(i(i+1)) to the i-th step-size (with i as in Eq. (9)), and set the expert learning rate to alpha = 1/T^{1/2}. The regret bound for the exponential weights wrt i is of order T^{1/2} log (i(i+1)) [more sophisticated experts algorithms would yield (T log (i(i+1)))^{1/2} regret wrt to i, but this does not bring any improvement here], which after substituting for the optimal i^* = O(log P_T) for a given value of P_T, gives a total O( (T (1+P_T))^{1/2} + T^{1/2} log log P_T ) = O ((T (1+P_T))^{1/2}) universal dynamic regret bound.  A few minor corrections: - Line 113: "an" --> "a"; - Line 142: "holds" --> "to hold"; - Line 171: "setting" --> "set"; - It may be worth adding, probably around or after Eq (10) and (11), that regret wrt true losses f_t is lower than regret wrt surrogate losses \ell_t, with a line that simply shows it (e.g., f_t (x_t) - f_t (u_t) \leq \ell_t (x_t) - \ell_t (u_t) from (10) and (11)). Although, the current formulation is already clear and the trick is simple and standard enough; - Line 202: a reference to the relevant section and theorem can be added.   [1] M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML), pages 928–936, 2003.  [2] T. van Erven and W. M. Koolen. Metagrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29, pages 3666–3674, 2016.  [3] E. C. Hall and R. M. Willett. Dynamical models and tracking regret in online convex programming. In Proceedings of the 30th International Conference on Machine Learning, pages 579–587, 2013.


Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation
In this paper we present a hybrid active sampling strategy for pairwise preference aggregation, which aims at recovering the underlying rating of the test candidates from sparse and noisy pairwise labelling. Our method employs Bayesian optimization framework and Bradley-Terry model to construct the utility function, then to obtain the Expected Information Gain (EIG) of each pair. For computational efficiency, Gaussian-Hermite quadrature is used for estimation of EIG. In this work, a hybrid active sampling strategy is proposed, either using Global Maximum (GM) EIG sampling or Minimum Spanning Tree (MST) sampling in each trial, which is determined by the test budget. The proposed method has been validated on both simulated and real-world datasets, where it shows higher preference aggregation ability than the state-of-the-art methods.

This paper proposed a novel hybrid active strategy for collecting pair-wise samples for BT model estimation. The estimation is based on minimum spanning tree and several other insights. The proposed method shows reliable and more precise performance on simulated and real-world datasets. The paper is overall well organized.   - missing some prior arts in the active learning and BLT model fields. Just to recall a few from top of my head. http://www.jmlr.org/papers/volume17/15-189/15-189.pdf http://proceedings.mlr.press/v28/wauthier13.pdf https://arxiv.org/abs/1109.3701 - There has been a lot of abbreviations in the paper which makes it difficult to read.  - In table 1, the author claimed that  the computation cost is comparable with Hodge based methods in large-scale crowd-sourcing experiments. It is not clear why this is a valid claim and is quite opposite from what stated in table 1.   [based on the author response and the re-organized/commented computation cost table, it is clear that how the proposed approach gain advantage in terms of computation cost. It would be good to update the caption as well as the discussion in the paper draft to make this confusing point clear. In addition, since there are some "navie" non-active baselines, it would be good to clarify in the caption of table saying these approaches are not appropriate to compare and judge efficiency. ]  Some additional comments: - In section 3.2, it would be better to clarify how the ground-truth labels/scores are assigned so it is easier to understand the RMSE/PLCC metrics.  Though there is a space limits, some high-level description might help reader to understand in these two problems, what are the target problem and why intuitively ranking them are challenging tasks.   - It seems a bit untypical that section 2.3.3 contains simulation while other experiments are in separate sections. Many of the simulation settings are discussed here which kind of complicated the reading process.   - it would be better not to use many abbreviations in the main algorithm. Indeed, in Alg 1 the main steps of estimating info-gain eq(6) is not what used in the end. It is explicitly eq(7) and its practical approximations that enables the efficient computation. Highly suggest to make Alg. 1 referring to the exact algorithms used.   -2.0 section is labeled as "notation" but actually contains key distribution assumptions.   - Figure 1 (a) is almost unrecognizable. Please update it and include the parameters used to simulate these contour plots.


Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing.

This paper introduces a simulation framework -- called a pseudoreality -- that can test an entire autonomous driving system, including deep-learning perception + control algorithms, underlying dynamics models and a photo-realistic rendering engine. They demonstrate full-scale testing using a risk-based framework, evaluating the probability of an accident under some base distribution that stipulates traffic behavior. Adaptive importance sampling methods are used to evaluate risk. The authors demonstrate that system testing can be done by 10-50 times the number of processors used in real-world testing and 1.5-5 times faster than that of naive MC sampling.  The problem domain examined in this paper is one that is extremely relevant, given the introductions of autonomous vehicles onto roads, both in city and highway driving. Due to the rare nature of accidents and uncertainty in real-world environments, testing is computationally prohibitive and so some verification in simulated environments is needed to ensure safety and performance guarantees. A probabilistic "risk-based" framework that simulates rare events only makes sense. The authors try to estimate the probability of a rare event via simulations and generate scenarios that exhibit risky behavior (defined by a threshold parameter).  The key premise of the proposed simulator requires that a model of the world be estimated. This is done by using public traffic data collected by the US DOT (Department of Transportation). Imitation learning is used to learn a generative model for the behavior/policy of environment vehicles, resulting in an ensemble of models that characterize the distribution of human-like driving policies. Specifically, an ensemble of generative adversarial imitation learning models are used to learn the base distribution.  To generate risky scenarios and to estimate the probability of a risky event, the authors use adaptive importance sampling (instead of naive MC sampling) to speed up evaluations. They propose a cross-entropy algorithm that iteratively approximates the optimal importance sampling distribution. Importance sampling + cross-entropy methods choose a distribution over the model parameters in the generative model of human driving behavior. The simulator then constructs a random rollout of behaviors according to the learned policy of a human driver given environmental inputs.  The simulator, then, is the primary contribution of this paper, which is supported by the risk-based framework and cross-entropy search techniques. It can take inputs such as an rendered image and output actuated commands based on an end-to-end highway autopilot network. Testing of the simulator is done using a multi-agent highway scenario where public traffic data is available. The authors show that their simulator accelerates the assessment of rare-event probabilities with respect to real-world testing.   The ideas presented in this paper are interesting in that they attempt to address the problem of how to test and ensure safety and performance of autonomous driving vehicles. Combining adaptive importance sampling + cross-entropy methods in a simulator that attempts to estimate and generate risky events seems like a reasonable approach.  From the paper's exposition, it appears that this is a novel idea and, if so, would be certainly of interest to the machine learning community.  I have read the authors' response and thank them for their comments.