The Physical Systems Behind Optimization Algorithms
We use differential equations based approaches to provide some physics insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's methods as well as their Nesterov's accelerated variants in a unified framework motivated by a natural connection of optimization algorithms to physical systems. Our analysis is applicable to more general algorithms and optimization problems beyond convexity and strong convexity, e.g. Polyak-Łojasiewicz and error bound conditions (possibly nonconvex).

1. Summary  This paper described the connection between various optimization algorithms for convex functions and similar variants and differential equations that describe physical systems. This work builds on previous work using differential equations to analyze optimization algorithms.   2. High level paper  I think the paper is a bit hard to read from Section 3 onward. Section 3 is packed with proofs of convergence for various types of functions, and new notation is frequently introduced. The contribution of the paper seems novel. And I think this work would be interesting to the optimization community.  3. High level technical  Overall I think the biggest things that could be improved are organization of the paper, and the clarity of the writing. Organization: I would spend more time explaining the proof steps in Section 3 and the general connection between optimization algorithms and physical dynamics in Section 5. I think the most interesting contribution of this paper is this optimization-physics connection. This paper will have a lot more impact if you take the time to deeply explain this connection, rather than go through the proofs of each convergence rate. For instance, it would be great to answer the questions: Why must the potential energy decrease for an algorithm to converge? Why is connection between the PL condition and Hooke's law important, or what insight does it give us? Why is the PL condition useful (I know that you cite [4] for examples, but I think you should include some to help with motivation). To have more space for all this I would move Section 4 to the appendix. And possibly move some proofs in Section 3 as well. Clarity: Right now Section 5 assumes a lot of background knowledge in physics. With the additional space I would describe these concepts in more detail, similar to Section 2. A few more specific comments that could improve clarity: - Lines 37-38: Could you be more specific when you say 'analyses rely heavily on algebraic tricks that are sometimes arguably mysterious to be understood'? This would better motivate the method. - Equations above 2.4 (Taylor expansion): Could you derive this quickly? - Line 115: Is it reasonable to require c -> c_0? Doesn't this mean that \eta = O(h) always? Is this reasonable? It would be good to discuss this. - Proof of Theorem 2.1: What does 0^+ mean? - Equation (3.1): How does Theorem 2.1 imply the bound? Do we assume that \gamma(0^+)=1 and f(X(0^+))=0, and do we ignore \gamma(t)? If so, then I see how Theorem 2.1 implies eq. (3.1), but otherwise I don't... - Line 177: It only implies this if m=0 or X^{..}=0, is this true?  4. Low level technical  - Line 1: 'differential equations based' -> 'differential equations-based' - Line 17: 'that achieves' -> 'that it achieves' - Line 39: 'recently attract' -> 'recently have attracted' - Line 45: 'are lack of link' -> 'lack a link' - Line 90: 'We starts with' -> 'We start with' - Line 124: I would remove the word 'mechanic' because this quantity is really the potential energy, as you describe below, and I think 'mechanic' creates confusion. - Line 156: 'independent on' -> 'independent of' - Line 167: 'strongly' -> 'strong' - Line 202: 'suffice' -> 'sufficient' - Line 270: 'As have been' -> 'As has been'  5. 1/2 sentence summary  Overall, while the paper is somewhat unclear, I think the novelty of the contribution, and its impact, justifies this paper being accepted.  Post-Rebuttal ---------------------- The authors do a very thorough job of addressing my review, but are slightly sloppy answering one of my questions: the response on lines 30-32 is a bit sloppy because: (a) they define t=kh in the paper, but in the response they say k=th, (b) in the response they say (x^{(k-1)} - x^{(k)}) = X(t + h) - X(t), but it must actually be equal to X(t - h) - X(t), furthermore, the initial equation (x^{(k-1)} - x^{(k)}) appears in none of the lines directly above equation (2.4) in the paper.  For me, I think this paper is worth accepting, but it is so dense that I think the authors need to move some of the proofs out of the main paper and into the supplementary to properly explain all of the concepts. Ultimately, because of their detailed response in the rebuttal, I still believe the paper should be accepted.


Learning in Games with Lossy Feedback
We consider a game-theoretical multi-agent learning problem where the feedback information can be lost during the learning process and rewards are given by a broad class of games known as variationally stable games. We propose a simple variant of the classical online gradient descent algorithm, called reweighted online gradient descent (ROGD) and show that in variationally stable games, if each agent adopts ROGD, then almost sure convergence to the set of Nash equilibria is guaranteed, even when the feedback loss is asynchronous and arbitrarily corrrelated among agents. We then extend the framework to deal with unknown feedback loss probabilities by using an estimator (constructed from past data) in its replacement. Finally, we further extend the framework to accomodate both asynchronous loss and stochastic rewards and establish that multi-agent ROGD learning still converges to the set of Nash equilibria in such settings. Together, these results contribute to the broad lanscape of multi-agent online learning by significantly relaxing the feedback information that is required to achieve desirable outcomes.

This paper certainly seems to contain some new insights and interesting results but my main complaint is that it is so ill-placed in context and literature which makes it a tough read for both the expert and non-expert. On the one hand I think the authors miss really a lot of literature both in their introduction and related work section (see below), and on the other hand (more importantly) I think there are some recent related works that are not referred to or discussed, which make it hard for the reader to grasp what now exactly the main contribution of the work is and how much is already covered in the literature.   Comments:  - The references to MAL in general in the introduction are fairly weak and oversees quite a number of important publications; the authors should give credit to the right publications where these are due. Also note that the overview paper is quite outdated and more recent overviews are available, thinking of Hernandez-Leal, Tuyls & Weiss, Bloembergen et al. (JAIR), Shoham et al. (and several of the response papers in AIJ) etc.  There is also a lot of literature on joint learning dynamics which is simply not referred to (see Bloembergen et al., Lanctot et al.)  - Related work also need a lot more references and needs to make the connection to regret-based dynamics such as the work of Klos et  al. (ECML), Lanctot et al., and e.g. NFSP etc.  - Fictitious play and n-regret have been very clearly linked (see e.g. Viosat et al., Jofani et al., Cannot and Pilouras), papers that really need to be referred and discussed; there is overlap with the work in this paper and it is not discussed.  - Some of the argumentation in the intro certainly makes sense, but a large part of the literature seems to be overlooked, and these works need to be discussed and given credit for.  - in the intro maybe good to say you don’t consider stochastic rewards.  - The reweighed online GD trick, by reweighing with probability p_i, defined as the expected value over the indicator value is not well explained. One weakness though is that it assumed the p_i’s are known (tackled in the last section though).  - The definition of the energy function really lacks some intuition; it requires an effort to follow the discussion from here on due to a lack of description of the ideas of the author(s).  - The main contributions in section 4 look nice, but I am unsure about all the the technicalities due to the compressed nature of writing this down (the supplementary material provides more clarity though)  - Update after reading author response: I am happy with the authors response but do expect them to make the changes committed to.


Dual Principal Component Pursuit: Improved Analysis and Efficient Algorithms
Recent methods for learning a linear subspace from data corrupted by outliers are based on convex 1 and nuclear norm optimization and require the dimension of the subspace and the number of outliers to be sufficiently small [27]. In sharp contrast, the recently proposed Dual Principal Component Pursuit (DPCP) method [22] can provably handle subspaces of high dimension by solving a non-convex 1 optimization problem on the sphere. However, its geometric analysis is based on quantities that are difficult to interpret and are not amenable to statistical analysis. In this paper we provide a refined geometric analysis and a new statistical analysis that show that DPCP can tolerate as many outliers as the square of the number of inliers, thus improving upon other provably correct robust PCA methods. We also propose a scalable Projected Sub-Gradient Method (DPCP-PSGM) for solving the DPCP problem and show that it achieves linear convergence even though the underlying optimization problem is non-convex and non-smooth. Experiments on road plane detection from 3D point cloud data demonstrate that DPCP-PSGM can be more efficient than the traditional RANSAC algorithm, which is one of the most popular methods for such computer vision applications.

This work has provided an improved analysis for global optimality of the (nonconvex) dual PCA problem based on random model assumption, and a more efficient optimization method based sub-gradient descent with a local linear convergent rate. The paper is well-organized overall, and the theoretical improvement is substantial compared to the existing work. The explanations of analysis in Section 2 and algorithms in Section 3 are not very clear and can be improved in the future. Below are some more detailed comments:  1. Assumptions of the random model: First, aside from the benefits of the analysis, how does the random spherical model of the subspace relates to applications? Does real data (both outliers and subspace) follow random spherical distribution? The reviewer feels that more validation is needed. Second, the description of the random model of the subspace in Theorem 2 is not very clear. From the reviewer’s understanding, is it that the columns of O is uniformly drawn from the sphere S^{D-1}, and the columns of X is uniformly drawn from a lower dimension sphere that S^{D-1} \cap A, where A is and dimensional subspace? The current description is not very clear and quite misleading. Besides, is there any assumption for the properties of the subspace A?   2. The relationship between M and N. From Theorem 2, if the reviewer understand correctly, the authors hid the dimension D  and d in the relation of M and N. If the dimension D and the subspace dimension d are counted, it should be M<=O(d/D*N^2), which makes sense as the subspace dimension d (ambient dimension D) gets bigger, more (fewer) outliers are tolerated. As the dimension D could be very large for many applications, the reviewer feels those parameters (d and D) cannot be ignored and should be accounted and explained in the introduction and the remarks.  3. For the subgradient method, could the author explain the intuition behind the choice of the initialization? Why does such a choice produce an initialization with small principle angle? On the other hand, for the subgradient method, does the linear convergence of subgradient method mainly due to the choice of the step size or due to the structure of the problem? For general convex nonsmooth problems, the convergence rate of subgradient method is usually sublinear. The reviewer feels that the reason could be that the local region around the optimal solutions has certain benign geometric structures (i.e. satisfies certain regularization conditions) that make linear convergence possible. It would be better if the author provides better explanation of this ``surprising’’ algorithmic property.  4. Possible Extensions: (1) The current work only considers finding a single subspace, does the current analysis possible to be extended to multiple (orthogonal) subspaces (e.g. using deflation or optimization over oblique manifold)?  (2) Instead of l1 norm loss, the author could consider a Huber loss as analyzed in [21], which is first order smooth which is much easier to analyze.  5. typos: Riemann gradient => Riemannian gradient


Proximal SCOPE for Distributed Sparse Learning
Distributed sparse learning with a cluster of multiple machines has attracted much attention in machine learning, especially for large-scale applications with high-dimensional data. One popular way to implement sparse learning is to use L 1 regularization. In this paper, we propose a novel method, called proximal SCOPE (pSCOPE), for distributed sparse learning with L 1 regularization. pSCOPE is based on a cooperative autonomous local learning (CALL) framework. In the CALL framework of pSCOPE, we find that the data partition affects the convergence of the learning procedure, and subsequently we define a metric to measure the goodness of a data partition. Based on the defined metric, we theoretically prove that pSCOPE is convergent with a linear convergence rate if the data partition is good enough. We also prove that better data partition implies faster convergence rate. Furthermore, pSCOPE is also communication efficient. Experimental results on real data sets show that pSCOPE can outperform other state-of-the-art distributed methods for sparse learning.

This paper mainly focused on solving regularized finite-sum problem in distributed setting. The first contribution is that it developed pSCOPE, which is an improvement over an existing SCOPE by combining proximal mapping to deal with non-smooth regularization term. Then a new measure to determine the quality of data partition is introduced, and the convergence of pSCOPE is analyzed with this measure. Furthermore, a trick for dealing with sparse data efficiently for pSCOPE is also proposed. Finally, some numerical comparisons were conducted to show the efficiency of pSCOPE method.  I do not think turning an existing method SCOPE into the proximal version is quite interesting or has great contribution. Though the authors listed several differences between SCOPE and pSCOPE, I still think these differences are tiny. On the other hand, I personally prefer the part about how to measure the quality of data partition and how it affects convergence rate, which should be novel. However, it is pity that such theory was not be utilized to guide the data partition in the paper.  The convergence rate is in fact poor, which has a quadratic dependence on the condition number. I do not agree this quadratic dependence is due to distributed computation. The effect of distribution should be reflected through $\xi$ instead of $\kappa$ (note that even for p=1, you still have $\kappa^2$). While most SVRG variants, no matter distributed or non-distributed, do have linear dependence on condition number.   Some experiment results look strange for me. But the convergence behavior of the baselines in some figures are really weird. Some of them seem not convergent at all. I am not challenging authors’ experiment here. I sincerely suggest the authors can recheck their experiments carefully, especially the implementation of these baselines.  There also exist several formatting issues in the paper. For example, the math equation should be more professional, especially the proximal mapping operator. And The figures are really too small for publishing.  I believe in line 162 that the square of the norm is missing.  Based on the above facts, I mainly remain neutral but slightly tent to accept this paper.   *********************************************************************** I have read authors’ feedback. The explanation to why the convergence has quadratic dependence on condition number is reasonable now. However, I still prefer a bound with linear dependence on condition number, since in most applications we do have individual convexity. Hence, I do not plan to change the score.


Integrated accounts of behavioral and neuroimaging data using flexible recurrent neural network models
Neuroscience studies of human decision-making abilities commonly involve subjects completing a decision-making task while BOLD signals are recorded using fMRI. Hypotheses are tested about which brain regions mediate the effect of past experience, such as rewards, on future actions. One standard approach to this is model-based fMRI data analysis, in which a model is fitted to the behavioral data, i.e., a subject's choices, and then the neural data are parsed to find brain regions whose BOLD signals are related to the model's internal signals. However, the internal mechanics of such purely behavioral models are not constrained by the neural data, and therefore might miss or mischaracterize aspects of the brain. To address this limitation, we introduce a new method using recurrent neural network models that are flexible enough to be jointly fitted to the behavioral and neural data. We trained a model so that its internal states were suitably related to neural activity during the task, while at the same time its output predicted the next action a subject would execute. We then used the fitted model to create a novel visualization of the relationship between the activity in brain regions at different times following a reward and the choices the subject subsequently made. Finally, we validated our method using a previously published dataset. We found that the model was able to recover the underlying neural substrates that were discovered by explicit model engineering in the previous work, and also derived new results regarding the temporal pattern of brain activity.

In this paper the authors use a recurrent neural network model to study reward processing in human fMRI experiments. The main novelty of this approach is that it takes into account both the neural data (measured by fMRI) and the behavioral outcomes. In contrast the standard approach is to fit a computational model on the behavioral data, then use that as a regressor in a classical GLM analysis, which may find a good solution for the behavioral data but one that is different from the solution implemented in the brain. In this respect, this application stands out. Most other neural network models that are used to model fMRI data (e.g. Guclu and van Gerven, J Neuroscience 2015) do not use the fMRI data to train the neural network directly.  The paper is well-written and clearly explained. The network topology the authors use is simple, elegant and appropriate for the proposed application. The implementation is fairly careful and the proposed method to 'unroll' the temporal dynamics to find the contribution of each timepoint and brain region to the reward signal (eq 11) is a nice way to visualise the underlying dynamics of the model, although not strictly novel because I am aware of other methods that use the derivatives of the predictive model for localising the discriminative signal.   As such, I find this work to be highly original, very clearly written and of a sufficient methodological quality to be published at NIPS, contingent on a satisfactory response to the questions outlined below. I also think that the method could be adapted for other problems relatively easily so does make a significant contribution. My comments for improvement mainly relate to clarity:  I would suggest to split the figures 3 and S1 and try to include all of them in the main manuscript (rather than splitting across main text and supplementary material). Switching from the different components in panels c and d of both respectively was initially quite confusing and I had to read the descriptions of both in the main text several times before I understood them. This could definitely be improved.   The procedure for parameter optimisation part was not really clear (e.g. it is not really clear how Figure S2 is involved in parameter optimisation), nor is it clear which different settings for lambda (eq. 7) were evaluated. Was this done via a grid-search? How was leave-one-out cross-validation performed (e.g. subject level?). The authors state 'see the supplementary material for details of the optimization methods' but this only seems to refer to the optimizers chosen, not parameter optimization per se.  Also it appears that a single cross-validation loop was employed, then the same parameter setting was used for all subjects. Is that correct? Please provide more details so that the reader can determine whether proper, unbiased parameter optimization techniques were employed.


Self-Erasing Network for Integral Object Attention
Recently, adversarial erasing for weakly-supervised object attention has been deeply studied due to its capability in localizing integral object regions. However, such a strategy raises one key problem that attention regions will gradually expand to non-object regions as training iterations continue, which significantly decreases the quality of the produced attention maps. To tackle such an issue as well as promote the quality of object attention, we introduce a simple yet effective SelfErasing Network (SeeNet) to prohibit attentions from spreading to unexpected background regions. In particular, SeeNet leverages two self-erasing strategies to encourage networks to use reliable object and background cues for learning to attention. In this way, integral object regions can be effectively highlighted without including much more background regions. To test the quality of the generated attention maps, we employ the mined object regions as heuristic cues for learning semantic segmentation models. Experiments on Pascal VOC well demonstrate the superiority of our SeeNet over other state-of-the-art methods.

Self-Handicapping Network for Integral Object Attention  This paper proposes a new strategy to exploit image background information in attention networks. The use of background priors in semantic segmentation improves the estimation of attention maps. The strategy proposes to exploit a self-building ternary mask consisting of an attention, a background, and a potential zone. To maintain such mask, conditional ReLUs are proposed, which switch activations between positive and negative values depending on a mask status. This is shown to produce better contrasts between potential zones from background areas. Technically, the framework builds a 3-branched network that refines an initial attention map. Experiments are illustrated with a weakly-supervised semantic segmentation. Results indicate an improved segmentation (intersection-over-union) over recent state-of-the-art approaches, using the Pascal VOC dataset.   The paper reads well, provides an exhaustive literature review on weakly supervised learning, focusing on attention network. Practical value of paper has direct high potentials in computer vision applications. Experiments demonstrate superiority of approach with respect to recent state-of-the-art. No obvious flaws in manuscript.   Possibly on the negative side, performance is higher but arguably on the same range of competitive approaches, perhaps showing a significance test on the segmentation results could give a stronger argument. If correct, "self-handicapping" means here "self-erasing". This latter terminology may perhaps be better related with the current literature on adversarial erasing methods? (I am aware of "erasing" networks, but not on any "handicapped" networks)  The authors' feedback proposes to add a figure with best and worst visual results - taking half a page - is a 9th page allowed? if not, the rebuttal loose its value - Furthermore, as mentioned in the review, results are in the same range as competing methods - authors missed addressing this question, on whether the proposed results consistently outperform other method (statistical significance of their results) - they only provides average accuracies - unless this was misunderstood from my side?


Spectral Signatures in Backdoor Attacks
A recent line of work has uncovered a new form of data poisoning: so-called backdoor attacks. These attacks are particularly dangerous because they do not affect a network's behavior on typical, benign data. Rather, the network only deviates from its expected output when triggered by a perturbation planted by an adversary. In this paper, we identify a new property of all known backdoor attacks, which we call spectral signatures. This property allows us to utilize tools from robust statistics to thwart the attacks. We demonstrate the efficacy of these signatures in detecting and removing poisoned examples on real image sets and state of the art neural network architectures. We believe that understanding spectral signatures is a crucial first step towards designing ML systems secure against such backdoor attacks.

This paper provides a methodology called spectral signatures that analyzes the covariance of the feature representation learnt by the neural network to detect backdoor attacks. Detecting backdoor attacks in neural networks is an important and very recent problem that hasn't been solved properly. The paper is well-written and easy to follow. However, I found that the evaluation is limited and there are some inconsistencies in the paper.   After reading the rebuttal, I am still concerned by the limitations in the experimental evaluation. Namely, the authors did not test to see whether the algorithm would be tricked by benign outliers or strongly multimodal distributions. While Great Danes are quite different from chihuahuas, there is somewhat of a spectrum of dog breeds between them that would not really make chihuahuas outliers in the data. In other words, we agree that the distribution has high variance but not that it is multimodal or contains subpopulations in that sense. Additionally, they did not consider how an adversary may try to circumvent the defense. All in all, we still feel that it is just below the threshold because the authors haven't really shown that the method is robust to different types of data and/or poisoning attempts.   • The proposed methodology assumes that a single unpoisoned class does not contain subpopulations (Def. 3.1) the methodology relies on the ability of identifying differences in the means of subpopulations in the same class (poison vs. legitimate). However,  classification tasks may include multiple subpopulations in the same class e.g., mammals may contain bears, dogs, cats. I suggest that the authors to address this particular concern and clarify when their methodology works. Ideally, some experimental results should be provided if they claim their solution works for the above example. • The authors contradict themselves in the experimental results reported:   o In line 112: "With as few as 250 (5% of a chosen label) poisoned examples, we successfully achieve both of the above goals on the CIFAR-10 dataset." o In line 258: "As an aside, we note that 5% poisoned images is not enough to capture the watermark according to our definition in our examples from Figure 2, but 10% is sufficient." • The paper calls the backdoor a "watermark". A backdoor is not necessarily a watermark. For example, in the original paper by Gu et al. (2017) that introduced backdoor attacks, the poisonous images of stop signs contained very visible stickers on them. Furthermore, a watermark has a different meaning in the security community and it threw me off a little while I was reading. I suggest changing watermark to backdoor or backdoor trigger throughout the paper.  • For readability purposes, it would be good to label columns 4th and 6th in table 2.  • Finally, using a single dataset and network architecture to test the approach is a bit limiting to fully generalize the results.


Latent Gaussian Activity Propagation: Using Smoothness and Structure to Separate and Localize Sounds in Large Noisy Environments
We present an approach for simultaneously separating and localizing multiple sound sources using recorded microphone data. Inspired by topic models, our approach is based on a probabilistic model of inter-microphone phase differences, and poses separation and localization as a Bayesian inference problem. We assume sound activity is locally smooth across time, frequency, and location, and use the known position of the microphones to obtain a consistent separation. We compare the performance of our method against existing algorithms on simulated anechoic voice data and find that it obtains high performance across a variety of input conditions.

This paper describes a system for localization and separation of sound sources recorded by distributed microphone arrays.  It is similar to MESSL (Mandel et al, 2009) in its building blocks, but builds upon it by employing a Gaussian process prior to encourage smoothness in time, frequency, and location.  It also is targeted to the same application as DALAS (Dorfan et al, 2015) of microphones being distributed in pairs around a large space, but adds the smoothness over time and frequency via the Gaussian process prior.  It is evaluated on synthetic sound mixtures that are anechoic, but contain a reasonable amount of additive uncorrelated noise.  Its performance is quantified using a new metric that compares the masks it estimates at each microphone to the ideal ratio mask at each microphone.  Results show that it generally provides a higher level of "power precision" compared to other systems while achieving a similar level of "power recall".   Quality:  This paper is thorough in its description, derivations, and evaluation.  The targeted situation is a little contrived, but the same system seems like it should be able to apply straightforwardly to other situations.  The approach is sound and introduces a very nice and flexible framework for incorporating prior assumptions into this model.  Evaluation is carried out carefully and shows good results for the proposed system.  One relatively minor shortcoming of the paper is that it is not clear that new metrics are necessary for evaluating the performance of this system, and make it slightly harder to contextualize the results.  This is especially true with the introduced precision/recall-style metrics which do not provide a single figure of merit with which to compare systems.  There are several methods that would be appropriate for comparing synthetic multi-channel inputs and outputs, such as BSS_EVAL or PEASS.  Single-channel metrics such as STOI might also be used by finding the best microphone signal to analyze, as was done with the MESSL results in the current paper.   There are a improvements to the overall approach that could be addressed in subsequent work:  It would also be interesting to use the derived masks to perform beamforming, which would produce a single-channel estimate of each source that could be compared to the original using STOI or subjective evaluations.  It would be more realistic to use diffuse noise rather than noise that is uncorrelated across sensors.  Diffuse noise is correlated at low frequencies, see for example  Habets and Gannot (2007).   Habets, E. A., & Gannot, S. (2007). Generating sensor signals in isotropic noise fields. The Journal of the Acoustical Society of America, 122(6), 3464-3470.    Clarity:  It is well written and generally easy to follow the logic, although many of the details have been compressed to save space, especially in the model description and derivation in section 2.  Since the paper provides a sophisticated combination of graphical modeling with microphone array processing, such details could be useful for readers from each of those communities to better understand the other, but it is understandable given the page limit.  One specific point that could improve the clarity would be describing the details of the kernels used in equations (1) and (2) adjacent to those equations instead of in the experiments section.  The baseline algorithm also performs surprisingly well, for example in figure 2.  It would be useful and interesting to describe it in more detail.   Minor clarity issues/questions: * On line 138, should C have a subscript associated with it, i.e., should there be different C's for different sources? * On line 141, why is only beta normalized? * On line 149, in most of the paper t_1 appears to be a time index at the frame level, but here it appears to be at the sample level, could this be clarified? Is this smoothing happening between frames? What w(.) function was used in the experiments?    Originality:  The paper does a good job of describing related work and directly comparing it to the proposed system.  In fact, some description of the previous work could perhaps be compressed, since it is discussed in the main introduction, the Prior Work section, and the Experiments section.  The proposed approach provides a very nice framework for extending MESSL that can be built upon in the future with additional assumptions and components.    Significance:  The proposed system provides a interesting improvement of the MESSL approach and paves the way for further improvements.  Empirically it seems to show good performance in the setting in which it was evaluated.   Response to author feedback:  I agree with much of what the authors say in their response, but do not fully agree that SIR and SDR only focus on "precision" of source separation.  While there are several incompatible definitions of SIR and SDR in the literature, there are certain existing definitions, for example SNR_ME in [A] that combine both precision and recall by measuring the ratio of the target speech present in the output (precision) to the combination of target speech energy absent from the output (recall) and the noise energy present in the output.  BSS_EVAL, while widely used, does in fact focus on precision to the exclusion of recall.   [A] Mandel, M. I., Bressler, S., Shinn-Cunningham, B., & Ellis, D. P. (2010). Evaluating source separation algorithms with reverberant speech. IEEE Transactions on audio, speech, and language processing, 18(7), 1872-1883.


Coordinate Descent with Bandit Sampling
Coordinate descent methods usually minimize a cost function by updating a random decision variable (corresponding to one coordinate) at a time. Ideally, we would update the decision variable that yields the largest decrease in the cost function. However, finding this coordinate would require checking all of them, which would effectively negate the improvement in computational tractability that coordinate descent is intended to afford. To address this, we propose a new adaptive method for selecting a coordinate. First, we find a lower bound on the amount the cost function decreases when a coordinate is updated. We then use a multi-armed bandit algorithm to learn which coordinates result in the largest lower bound by interleaving this learning with conventional coordinate descent updates except that the coordinate is selected proportionately to the expected decrease. We show that our approach improves the convergence of coordinate descent methods both theoretically and experimentally.

The paper introduces a new adaptive coordinate descent algorithm, which selects the coordinates based on a measurement of marginal decrease. The main idea is to measure the marginal decrease of each coordinate with the help of dual residues. The greedy algorithm consists of choosing the coordinate which provides the largest  marginal decrease. However, the greedy version requires the information of all the coordinates which is computationally expensive. To overcome this issue, an adaptive variant which updates the information in an online style has been introduced.   My major concern is about the theoretical analysis of the adaptive algorithm. In Proposition 2, the convergence rate of the algorithm is given under the assumption that the initial function gap f(x_0)-f* is smaller than some quantity Q. However, this assumption is unnatural and inappropriate due to the dependance between Q and f. It is not true that this condition will hold by scaling down the function f, because Q will also scale down. Moreover, Q depends on another highly non trivial quantity c, which depends on the functional property and probably the initial point. Thus, a more careful clarification is needed.   Besides this negative point in the theoretical analysis, I find the experimental section unconvincing. In particular, the parameter choices seems very arbitrary, for example the regularization parameter. Moreover, it is not clear how the algorithm's key parameters are selected like \epsilon and E. For ridge regression, E is set to be n/2 and for other cases E is d/2. There might be a huge gap between the feature dimension d and the number of samples n. It sounds like taking the best value that works without any theoretical reason.  Overall, the idea of using the marginal decrease as a guidance of coordinate selection is very interesting, but I do not support publication of the paper in the current state due to the weakness in both theoretical and the experimental results.   Some additional remarks: 1) The l1 regularization is not L-bounded unless taking L=infinity  2) Some of the plots do not include any algorithm proposed in the paper, for example in logistic regression on dataset w8a or a9a in Figure 3.   EDIT: I thank authors for addressing may concerns. Even though I still find the theoretical result a bit weak, I believe the paper provides interesting insight and ideas with outstanding empirical performance, as such, I raise my score accordingly.


Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention for developing optimization algorithms with fast convergence. However, the studies of EBC in statistical learning are hitherto still limited. The main contributions of this paper are two-fold. First, we develop fast and intermediate rates of empirical risk minimization (ERM) under EBC for risk minimization with Lipschitz continuous, and smooth convex random functions. Second, we establish fast and intermediate rates of an efficient stochastic approximation (SA) algorithm for risk minimization with Lipschitz continuous random functions, which requires only one pass of n samples and adapts to EBC. For both approaches, the convergence rates span a full spectrum between O(1/ √ n) and O(1/n) depending on the power constant in EBC, and could be even faster than O(1/n) in special cases for ERM. Moreover, these convergence rates are automatically adaptive without using any knowledge of EBC.

Overview:  This paper develops fast excess risk bounds for ERM and develops stochastic optimization algorithms for losses that (at the population level) satisfy the so-called "error bound condition". The error bound condition generalizes a particular inequality implied by strong convexity to 1) non-convex but still curved losses 2) convex losses that are not strongly convex but enjoy similar (possibly slower) growth around the optimum. The authors give 1) An ERM guarantee that interpolates between \sqrt{d/n} and d/n as a function of the exponent for which the EBC holds, and does not assume convexity and 2) A faster/optimistic rate that depends on the loss of the benchmark, but requires convexity in addition to EBC 3) A stochastic optimization routine matching the (not optimistic) guarantee 1). Lastly, they instantiate their guarantees for a few simple families of losses.  Originality and Significance:  My takeaway is that the results in this paper do not require huge changes in proof technique compared to previous results that assume strong convexity/convexity (eg van Erven et al. "Fast rates in statistical and online learning" for Theorem 1 and Zhang et al., "Empirical risk minimization for stochastic convex optimization: O(1/n)- and o(1/n^2)-type of risk bounds" for Theorem 2), but I do like that the paper is fairly comprehensive and I think it will probably serve as a useful reference point for future research. Some further coments:  * Theorem 2 is fairly restrictive since it has to assume convexity, but I see from the examples that there are indeed losses that satisfy EBC for theta > 0 yet are not strongly convex.  * Theorem 3: To make the paper more comprehensive it would be better if this result could be extended to L*-dependent regime studied in Theorem 2. Also, please add more discussion as to why adapting to the parameter theta is a major technical challenge,   Quality and Clarity:  Overall I found they paper to be reasonably well-written and fairly easy to follow, though the intro has quite a few typos so please give it another pass. I do have some minor comments: * Line 42: Please be more precise on your assumptions on data and loss for the rates you give for ERM and SA. ERM can also get \sqrt{1/n} also depending on the assumptions on the data norm. * The PLP setup / (12) should apply to piecewise linear *convex* functions, correct? * Ass 2: Specify the norm wrt Lipschitzness is defined. * Line 107: The result of Srebro you mention can easily be adapted to the unknown L* case using the doubling trick, no?  ------------------ After rebuttal: I am keeping my score, as the rebuttal only addresses minor technical comments and not the overall issue of novelty in analysis and proof techniques.


DropMax: Adaptive Variational Softmax
We propose DropMax, a stochastic version of softmax classifier which at each iteration drops non-target classes according to dropout probabilities adaptively decided for each instance. Specifically, we overlay binary masking variables over class output probabilities, which are input-adaptively learned via variational inference. This stochastic regularization has an effect of building an ensemble classifier out of exponentially many classifiers with different decision boundaries. Moreover, the learning of dropout rates for non-target classes on each instance allows the classifier to focus more on classification against the most confusing classes. We validate our model on multiple public datasets for classification, on which it obtains significantly improved accuracy over the regular softmax classifier and other baselines. Further analysis of the learned dropout probabilities shows that our model indeed selects confusing classes more often when it performs classification.

This paper proposes doing dropout in the output softmax layer during supervised training of neural net classifiers. The dropout probabilities are adapted per example. The probabilities are computed as a function of the penultimate layer of the classifier. So that layer is used to compute both : the logits, and the gating for those logits. This model combines ideas from adaptive dropout (Ba and Frey NIPS'13) and variational dropout (Kingma et al).  The key problem being solved is how to do inference to get the optimal dropout probabilities. There are several approximations made in the model in order to do this inference. If I understood correctly, the model essentially comes down to the following:  Two gating functions are being imposed to gate the logits. One gating function is computed by taking the penultimate layer and passing it through learned weights (W_theta). The other gating function is also computed by taking the penultimate layer and passing it through learned weights (W_phi), but the second gating also tries to take the output y into account by regressing to it using an auxiliary loss. Together the two gating functions produce a number in [0, 1] which is used as the retain probability for doing dropout in the output softmax.  Pros  - The proposed model leads to improvements in classification performance for   small datasets and tasks with fine categories.  Cons  - The model appears to be overly complicated. Is just one gating function that   includes the dependence on y (W_phi) really not sufficient ? It would be great to see empirical evidence that this complexity is useful.  - The deterministic attention baseline seems to be using gating derived from x   alone (without the second gating used in the proposed dropmax method which also regresses to y). It would be useful to have a clear comparison between dropmax and a deterministic version of dropmax.  - The second gating function is essentially doing the same task as   classification but with a unit-wise cross-entropy loss instead of a multiclass cross-entropy loss which the main classifier is using. The resulting model is a product of the two classifiers.  It is plausible that the reported gains are a result of this ensembling of 2 classifiers, and not the dropout regularization. A determinstic baseline (as suggested above) would help resolve this question.  Overall : The model is presented as a new variant of dropout derived using a somewhat complicated setup, when it can perhaps be more simply understood as ensembling a unit-wise logitic regression loss and the familiar softmax loss. It is not clear if the current presentation is the cleanest way to describe this model. However, the model is sound and leads to incremental improvements.


Variance-Reduced Stochastic Gradient Descent on Streaming Data
We present an algorithm STRSAGA that can efficiently maintain a machine learning model over data points that arrive over time, and quickly update the model as new training data are observed. We present a competitive analysis that compares the suboptimality of the model maintained by STRSAGA with that of an offline algorithm that is given the entire data beforehand. Our theoretical and experimental results show that the risk of STRSAGA is comparable to that of an offline algorithm on a variety of input arrival patterns, and its experimental performance is significantly better than prior algorithms suited for streaming data, such as SGD and SSVRG.

This paper considers the problem of streaming stochastic optimization taking into account the arrival patterns of examples in time. Whereas the relevant previous work focuses on learning from a stream (i.e. in one pass over the data, with O(dimension) memory), this work attempts to utilize the time spent between the arrival of examples in order to revisit previously encountered examples.  The problem description is novel and appealing, both in its possible practical relevance and as an interesting new model for consideration in algorithm analysis. However, there are central issues with the comparisons that the paper draws between algorithms, both conceptually and experimentally.  Conceptually, it seems incorrect to say that STRSAGA is a streaming algorithm, and in turn in to repeatedly compare it to one (such as SSVRG), since STRSAGA's memory complexity actually grows linearly with the dataset size (in order to maintain its "effective sample set"). In all previous related work, a streaming algorithm essentially means one whose memory usage does not grow with the dataset size. This is exactly what distinguishes streaming algorithms from ERM-based algorithms. The latter maintain an empirical sample of points in memory, whereas the former cannot maintain more than a constant number of them.  Experimentally, the paper's main algorithm (STRSAGA) is missing some comparisons to natural baselines. Both DYNASAGA and SSVRG are not basic (nor widely used) algorithms. Meanwhile, a comparison to the simplest and most basic streaming algorithm, plain SGD, is missing.  Relating to baselines, the paper includes the introductory remark (line 74):  > We also show that STRSAGA significantly outperforms SSVRG [FGKS15], the state-of-the-art streaming data algorithm  The original SSVRG paper [FGKS15] offers no experiments, and I don't know of any published since. It's a theoretical result, being used here as an experimental baseline, as a means of demonstrating an empirical advantage. Because it isn't actually a commonly accepted state of the art in the practical sense, outperforming SSVRG in practice does not imply an improvement relative to the state of scientific knowledge. Again, the more obvious comparison here would be to plain one-pass SGD.  ----  Edited after author response:  The authors addressed the concern about experimental comparisons to SGD in their response. I suggest including this in the paper.  On the point of "streaming", here is a valid run of the algorithm:  Fix an integer k. Suppose there is a dataset of size 2k, and STRSAGA is run for steps 1, ..., k, where at each step it receives two subsequent data points from the dataset. After step k, STRSAGA has half of the original dataset stored in its effective sample set. Now suppose STRSAGA continues to run for T additional steps, receiving zero new points at each step. Then, during these T additional steps, STRSAGA processes that half of the original dataset that it had stored.  So STRSAGA's description includes, as a special case, an algorithm for minimizing empirical risk on half of a dataset, and this indicates that the paper needs work in its formal setup, not only in terminology. Maybe the right direction is to require a limit on memory, as reviewer 2 suggests as well.


Rectangular Bounding Process
Stochastic partition models divide a multi-dimensional space into a number of rectangular regions, such that the data within each region exhibit certain types of homogeneity. Due to the nature of their partition strategy, existing partition models may create many unnecessary divisions in sparse regions when trying to describe data in dense regions. To avoid this problem we introduce a new parsimonious partition model -the Rectangular Bounding Process (RBP) -to efficiently partition multi-dimensional spaces, by employing a bounding strategy to enclose data points within rectangular bounding boxes. Unlike existing approaches, the RBP possesses several attractive theoretical properties that make it a powerful nonparametric partition prior on a hypercube. In particular, the RBP is self-consistent and as such can be directly extended from a finite hypercube to infinite (unbounded) space. We apply the RBP to regression trees and relational models as a flexible partition prior. The experimental results validate the merit of the RBP in rich yet parsimonious expressiveness compared to the state-of-the-art methods.

The paper describes a process to generate (grid-aligned) boxes in spaces of arbitrary dimension. The authors show the process obeys many desirable/necessary properties in the construction of Bayesian nonparametric models.  In some regards, I would have fought for the paper to be accepted: I like how simply the model is described and motivated, and the experiments are wonderfully broad and thorough. But the paper makes serious errors on omitting necessary technical components that a paper such as this should have. For example, the model is briefly described in Sec. 3, but nowhere is a likelihood function (joint distribution over the model) provided! Neither are any descriptions of what the posterior distributions look like (or which ones we're interested in targeting), which leads me to my biggest criticism of the paper: The serious misjudgment of the authors to leave any details of inference out of the paper. I believe the only cases where inference details can be relegated to the supplementary material is when the algorithm used is a direct, unaltered application of an existing algorithm, and even then the paper should contain a high-level (technical) description. This is especially troubling for this paper since, as the authors themselves point out, this partitioning model is fundamentally different from those previously presented in the literature (this idea of bounding rather than cutting). Moreover, while reading the paper, I was very interested in the construction of the model, and in my head my main thought was "the most interesting part will be seeing how inference is performed." For example, sampling the start positions and lengths of each interval during the construction is beautifully simple, but when you have data in the grid, and given the position of one interval (on one axis), what does the updated distribution on the interval of a second axis look like? How would it be sampled? These are the most crucial (and meaningful) questions to ask when thinking about Bayesian approaches to inference. An even more sour taste is left in my mouth because the authors decided to instead keep Sec. 3.1--3.3 in the paper. Sure these are interesting results but it is unforgivable to have devoted so much to their description at the sacrifice of details of your inference procedure. I did read the details of inference in the supp (though I shouldn't have to) and it would have been far more useful to have that in the paper and even expanded, arduously.  This is a shame, your work has all the elements of being a great NIPS paper, and I would have fought for it. It really just comes down to what you chose to (and not to) write and how you've organized. I'll discuss with the AE.  More detailed comments:  p. 2, line 42: "budget" is never defined.  p. 2, line 69 "...restricted in discrete arrays.": I don't know what this means.  p. 3, line 94: I wouldn't say the ensembles of the trees are "easy-to-explain". Can you explain what a random forest learns?  p. 3, line 119: How do you sample P( l_k^{(d)} )?  p. 2 & 3: model description: Unless I missed it, I don't see what the "costs" are for. I am assuming that they somehow play a role in the likelihood and thus in the inference procedure...  Sec. 3.2 coverage probability: I'm confused... in what cases or types of construction would the coverage probability depend on the datapoint? IMO, it would be a very unnatural construction if it did.  Sec. 5 Experiments: This is great work. Very thorough and broad and beyond what I normally see in NIPS papers. I like the two demonstrative applications to relational modeling and regression trees. Some additional comments:  p. 6, Paragraph starting line 217: You really should put quick details of these datasets in the text. Such as size and some characterizing properties.  p. 8, paragraph starting line 259: This paragraph makes no sense to me. Read it a few times I still have no idea what it's saying.  p. 8, par starting line 267: What is \alpha?  Small complaint: I don't like the self-descriptions of your model of "parsimonious" or "comprehensive". For example, in the abstract and intro and demonstrably p. 8, line 286. Can't you just say "visually intuitive (given evidence such as in Fig. 5)" or something like that?  Updated review:  My original score of 3 was a strong (and accurate) statement, voicing my opinions about what was not included in the paper, making it incomplete and inappropriate for acceptance. After reading the authors' response, I believe they agree with my assessment and will rectify this in the final version of the paper. If that is indeed corrected, then the paper is a good submission, justifying a score of 7 (in my opinion), which I stated in my original review. I will check back in on the final version and raise an objection to the AE and editors if this is these promises are not kept.


Discrimination-aware Channel Pruning for Deep Neural Networks
Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. In this paper, we investigate a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose those channels that really contribute to discriminative power. To this end, we introduce additional discrimination-aware losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels outperforms the baseline model by 0.39% in top-1 accuracy.

The paper presents an approach for channel pruning in deep neural networks. The authors propose to prune channels based on their "discriminative power" -- this is achieved by introducing additional per-layer losses, and performing channel selection and network parameter optimization in an iterative fashion. Experimental results on 3 datasets (CIFAR-10, ILSVRC-12, LFW) show that the method compares favorably with respect to the state of the art.  Overall, the paper is well written, and explanations are clear. The Supp. Material provides many further details / experiments. However, some parts of the text could be made clearer (e.g. : lines 47-53 pg. 2 became clear to me only after reading Sec. 3; notation in lines 135-140 pg. 4 could be simplified).  The experimental results show the effectiveness of the proposed approach, which seems to bring an original contribution. However, for some technical choices, it would be useful to get a better intuition/motivation. Did the authors experiment with different losses than (5)? Could it happen that some layers capture high-level features, which are difficult to evaluate based on the cross-entropy loss? It would be interesting to give insights about how the approach could be adapted to work for models trained with different (final) losses. Furthermore, it is unclear to me how the layers L_p (the ones with added losses) are selected. Is there any heuristic to decide how many per-layer losses should be added, and to which layers?  Finally, it would be useful to discuss how much computational effort/time the approach adds at training time.  Other notes / questions: - It would be interesting to discuss in more detail the reasons for the improvement observed on ILSVRC-12 after pruning. In some cases, did the authors observe better generalization capabilities for the model?  - It would be interesting to discuss if there are architectures for which the proposed approach is more/less suitable. - Does the quality of the pre-trained model heavily affect performances? - Sec. 3.2: When discussing limitations of reconstruction methods, it would be useful to add references supporting the claims. - When using acronyms, it would be better to write also the entire name, at the first occurrence (e.g. Stochastic Gradient Descent - SGD). - References: [10] appeared at ICCV, not CVPR. - Line 197, pg. 6: < instead of > ?


Multi-Class Learning: From Theory to Algorithm
In this paper, we study the generalization performance of multi-class classification and obtain a shaper data-dependent generalization error bound with fast convergence rate, substantially improving the state-of-art bounds in the existing data-dependent generalization analysis. The theoretical analysis motivates us to devise two effective multi-class kernel learning algorithms with statistical guarantees. Experimental results show that our proposed methods can significantly outperform the existing multi-class classification methods.

After author response: thank you for your detailed comparison.  ======  Proving generalization bounds is one of the most important tasks in theoretical machine learning. The current paper is focusing on multi-class classification where the output cardinality K is larger than 2. The investigation of multi-class classification has a long history (e.g., [1, 7, 12, 14]).  The paper claims to highly improve the generalization bound using the known technique of "local rademacher complexity" ([3]). This technique focuses on a smaller class that contains only hypotheses with small variance. The paper claims to improve the generalization bound from K/sqrt{n} to logK^(O(1))/n (where n is the sample size), which seems like an impressive improvement. Of course, this is under the assumption that the "true" hypothesis is with low variance.  It is unclear to me why the authors have decided not to reference and compare to the paper "Xu, Chang, et al. "Local rademacher complexity for multi-label learning." IEEE Transactions on Image Processing 25.3 (2016): 1495-1507" which has a similar goal: "We analyze the local Rademacher complexity of empirical risk minimization (ERM)-based multi-label learning algorithms, and in doing so propose a new algorithm for multi-label learning."


Variational PDEs for Acceleration on Manifolds and Application to Diffeomorphisms
We consider the optimization of cost functionals on infinite dimensional manifolds and derive a variational approach to accelerated methods on manifolds. We demonstrate the methodology on the infinite-dimensional manifold of diffeomorphisms, motivated by optical flow problems in computer vision. We build on a variational approach to accelerated optimization in finite dimensions, and generalize that approach to infinite dimensional manifolds. We derive the continuum evolution equations, which are partial differential equations (PDE), and relate them to mechanical principles. A particular case of our approach can be viewed as a generalization of the L 2 optimal mass transport problem. Our approach evolves an infinite number of particles endowed with mass, represented as a mass density. The density evolves with the optimization variable, and endows the particles with dynamics. This is different than current accelerated methods where only a single particle moves and hence the dynamics does not depend on mass. We derive theory and the PDEs for acceleration, and illustrate the behavior of this new scheme.

In this paper, the authors consider acceleration schemes when optimising functionals over Diff(R^n), the space of diffeomorphisms on R^n. The method is an analog to more classical approaches for of accelerated optimisation schemes on finite dimensional manifolds. The authors motivate the need to optimise functionals on Diff(R^n) by quoting applications to optical flows. The experiment section does a good job to study the effect of various hyper-parameters and set-up choices. Most experiments are toy experiments which make it easy to compare results with what one would expect from them. On the other hand, the only non-toy experiment is hard to analyse (it's really hard to tell if the found diffeomorphism is good, or even if it is better than the one found from normal gradient descent).  I enjoyed reading the paper, and found it handled well the treatment of very complicated mathematical objects. I nevertheless think that the paper would benefit from some more introductory references to some of these complicated subjects (see comments below).  Comments: 1. line 45, I don't get point 3. The kinetic energy is a functional, not a metric. Do the authors mean that this kinetic energy corresponds to a norm, and a Riemannian metric can be derived from this norm? But in that case, this is a family of metrics, since the kinetic energy (4) is time dependent: it depends on \rho, which is time dependent (see line 182). 2. line 124: don't you also need \gamma(0, s) = constant, and \gamma(1, s) = constant, in order to keep endpoints fixed. 3. line 163: this needs a reference here. There are many flavours of infinite dimensional manifolds (e.g.: Banach, Frechet). Also, with this definition of the tangent space, the kinetic energy (4) is not defined (in the sense of being finite) for all tangent vectors: just make $|v|$ go to infinity fast enough compared to $\rho$. 4. line 178: \rho should take values in R_+ 5. after line 183, formula (3): it would be clearer if \rho was replaced by rho_t everywhere in this formula, to make the time dependence clearer 6. after line 187, formula (4): it would be clearer to write $T(v, \rho)$ instead of $T(v)$ since this depends on \rho. Also, the integral subscript $\varphi(R^n)$ could be simplified to $R^n$ since $\varphi$ is a diffeomorphism of $R^n$. 7. after line 201, formula (6): I understand why this is equivalent to (2), but why is it better to use the inverse diffeomorphisms? 8. line 221: the sentence is mal-formed. 9. line 227: should $v$ be replaced by $v_\epsilon$ inside the limit? 10. line 264: the term 'reasonable' sized image is subjective. I'd suggest saying 'larger sized images' instead. 11. All experiments use a displacement that is smaller than the size of the squares. Is that because otherwise there is no gradient (as in, equals 0) if this condition is not satisfied. This would be good to clarify this. 12. All experiments use different values for $\alpha$, ranging from 0.02 to 8. Why change it so much, and how were these values chosen? 13. It would be good to have some visualisations of what diffeomorphisms was found by the method (for ex, for square translation, was this just a translation?). One possibility would be to create an image with some colour gradient along both the x and y directions, and show how the images look like after applying the diffeomorphisms. This would also help understand what sub-optimal solution was found by normal gradient descent for the 'square translation + non-uniform scaling' experiment. 14. The 2 MR cardiac images are extremely close to each other to start with. I personally see no qualitative difference between the solutions found by gd and agd. I also am not able to judge if any of these solutions is good. Having experiments on real data is good, but I don't find that this one is particularly useful. 15. In the Appendix, section E. Discretization, formula (20): $v_1\partial_{x_1}v_1$ should be $v_1\partial_{x_1}v_2$  ============  Thank you to the authors for their reply. After reading the other reviewers opinion, I will maintain my score and positive opinion of the paper, but I would like to emphasise a very valid point made by other reviewers: the term 'accelerated' is misleading. The algorithm proposed by the authors is inspired by 'accelerated' methods, but without theoretical bounds, I think this term should be used with more caution. In particular, I believe it should probably be dropped from the title. It can still be used in the paper when providing empirical evidence of acceleration, or when referring to other accelerated methods.


Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
It is widely believed that learning good representations is one of the main reasons for the success of deep neural networks. Although highly intuitive, there is a lack of theory and systematic approach quantitatively characterizing what representations do deep neural networks learn. In this work, we move a tiny step towards a theory and better understanding of the representations. Specifically, we study a simpler problem: How similar are the representations learned by two networks with identical architecture but trained from different initializations. We develop a rigorous theory based on the neuron activation subspace match model. The theory gives a complete characterization of the structure of neuron activation subspace matches, where the core concepts are maximum match and simple match which describe the overall and the finest similarity between sets of neurons in two networks respectively. We also propose efficient algorithms to find the maximum match and simple matches. Finally, we conduct extensive experiments using our algorithms. Experimental results suggest that, surprisingly, representations learned by the same convolutional layers of networks trained from different initializations are not as similar as prevalently expected, at least in terms of subspace match.

Overall I liked the method developed by this paper, it’s an interesting alternative to the SVCCA method of [1]. The paper is very well written, and displays a high level of mathematical rigor. My main disappointment is the authors do not do much with their new technique, the main experimental observation is the middle layers of two independently trained VGG networks are not similar compared to the bottom and top layers. This is intriguing and I’m really curious as to why, do the authors have any insight into this? I’d also be curious to interpret what very small simple matches represent. One could for example pick the max variance linear combination of neurons in a simple match and visualize the validation images by projection on this subspace, if the concept of a simple match were meaningful then we might be able to interpret this by visualizing the images sorted along these projections.   I’d also like to see experiments regarding a sort of “null hypothesis” for this sort of analysis. For example, what happens if we run these similarity measures to compare a trained network to an untrained network, maybe we would still see matches just by spurious correlations. Did the authors investigate this? How large does the maximum match need to be before we should conclude the layers are similar?  Also how does this method account for the possibility of completely distributed representations? That is the method looks for subsets of neurons between two networks that have the same span. However, one could imagine a setting where there is a particular linear combination of all neurons in X that have the exact same span of a different linear combination of all neurons in Y, but no strict subset of X and Y match according to the definition in this work. In this case one it seems reasonable to conclude that these two linear combinations compute the same representation (at least assuming the match holds on enough data points), but the proposed method might not pick this up if no strict subset of X and Y have the same span. Can the authors comment on this possibility?   A few typos, and other comments:   31 “understand”  267 “neural networks”   272: The conclusion that the lower layers are not similar based on the fact that high layer similarity only occurs at epsilon=0.5 seems a bit heuristic to me. How do we measure similarity of layers as a function of epsilon and layer similarity? Can the authors give some insight as to how to interpret the size of epsilon and maximum matching similarity?   https://arxiv.org/abs/1706.05806  Edit: In light of the rebuttal I'm changing my score to 7. I did not see all of the additional experiments in the Supplementary material on my reading, and found the plots visualizing the simple matches between independently trained networks to be compelling. Good work!


KDGAN: Knowledge Distillation with Generative Adversarial Networks
Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i.e., a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates. To address these limitations, we propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses. By simultaneously optimizing the distillation and adversarial losses, the classifier will learn the true data distribution at the equilibrium. We approximate the discrete distribution learned by the classifier (or the teacher) with a concrete distribution. From the concrete distribution, we generate continuous samples to obtain low-variance gradient updates, which speed up the training. Extensive experiments using real datasets confirm the superiority of KDGAN in both accuracy and training speed.

In this paper, the authors propose combining a knowledge distillation and GANs to improve the accuracy for multi-class classification.  (In particular, the GANs are most related to IRGAN, where the discriminator is distinguishing between discrete distributions rather than continuous ones.)  At the core, they demonstrate that combining these two approaches provides a better balance of sample efficiency and convergence to the ground truth distribution for improved accuracy.  They claim two primary technical innovations (beyond combining these two approaches): using the Gumbel-Max trick for differentiability and having the classifier supervise the teacher (not just the teacher supervise the classifier).  They argue that the improvements come from lower variance gradients and that the equilibrium of the minimax game is convergence to the true label distribution.  The idea of combining these two perspectives is interesting, and both the theoretical arguments and the empirical results are compelling.  Overall, I like the paper and think others will benefit from reading it.  However, I think there are a few gaps in the analysis that if closed would greatly benefit the paper: (1) Because there are a multiple design decisions made that are claimed to balance for an improvement, it is important to show that it is not one particular change that is causing the majority of the benefit.  In particular, using the classifier to jointly train teacher has been published in "Large scale distributed neural network training through online distillation" (which should be cited but is not).  Comparing KDGAN with co-distillation would be an interesting baseline (this is missed in the hyperparameter analysis because of the way alpha parameterizes the GAN loss).  I think this is important as co-distillation even without GAN will allow for the true label distribution to be a convergence point, while maintaining low-variance updates.  (1a) While the hyperparameter sweeps provide a useful analysis of where the benefits come from, another (though in my opinion less interesting) missing component is understanding how much benefit the Gumbel-Max trick provides. (2) Lower variance of KD than GAN: Section 3.3 claims that the gradients from the KD loss are lower variance than the gradients from the GAN loss.  While this is intuitively believable, and prior work in this area is cited as support, I don't see any proof or empirical evidence adding to this claim.  There are some slight notation improvements that could be made, such as being consistent as to whether D references the discriminator or the distillation (see line 195).  However, overall the paper is clear and the results are valuable.


Mental Sampling in Multimodal Representations
Both resources in the natural environment and concepts in a semantic space are distributed "patchily", with large gaps in between the patches. To describe people's internal and external foraging behavior, various random walk models have been proposed. In particular, internal foraging has been modeled as sampling: in order to gather relevant information for making a decision, people draw samples from a mental representation using random-walk algorithms such as Markov chain Monte Carlo (MCMC). However, two common empirical observations argue against people using simple sampling algorithms such as MCMC for internal foraging. First, the distance between samples is often best described by a Lévy flight distribution: the probability of the distance between two successive locations follows a power-law on the distances. Second, humans and other animals produce long-range, slowly decaying autocorrelations characterized as 1/f -like fluctuations, instead of the 1/f 2 fluctuations produced by random walks. We propose that mental sampling is not done by simple MCMC, but is instead adapted to multimodal representations and is implemented by Metropolis-coupled Markov chain Monte Carlo (MC 3 ), one of the first algorithms developed for sampling from multimodal distributions. MC 3 involves running multiple Markov chains in parallel but with target distributions of different temperatures, and it swaps the states of the chains whenever a better location is found. Heated chains more readily traverse valleys in the probability landscape to propose moves to far-away peaks, while the colder chains make the local steps that explore the current peak or patch. We show that MC 3 generates distances between successive samples that follow a Lévy flight distribution and produce 1/f -like autocorrelations, providing a single mechanistic account of these two puzzling empirical phenomena of internal foraging.

One of the current speculative hypotheses in cognition is that the brain performs approximate Bayesian inference via some form of sampling algorithm. Based on this assumption, this paper explores which kind of Monte Carlo algorithm the brain might be using. In particular, previous work has proposed direct sampling (DS) from the posterior distribution, or random-walk MCMC. However, these two algorithms are unable to explain some empirically observed features of mental representations, such as the power law seen in the distance between consecutive, distinct “samples” (such as responses in a semantic fluency task) or, equivalently under certain assumptions, of the distribution of inter-response intervals. Also, autocorrelation of mental samples exhibit a specific “1/f” pattern. This paper argues that another type of MCMC algorithm, that is MC3 aka parallel tempering, which is a MCMC method designed to deal with multimodal, patchy posteriors (hence the “foraging” analogy), is instead able to explain these features. The authors demonstrate the better empirical validity of MC3 -- mostly as a proof of concept -- with a toy example and with real data obtained in a simple novel experiment.  Comments:  Quality: The paper is technically sound and, if we accept the whole “mental sampling” paradigm, presents an interesting hypothesis for how sampling is performed (i.e., via parallel processes -- chains -- that operate at different “temperatures”, and occasionally swap information).  The current work is mostly a proof of concept in that the authors show general quantitative agreement with the data for some specific algorithm choices (e.g., temperature range, number of chains, proposal distributions). I think that a much broader exploration of the (hyper)parameter space is beyond the scope of this work (and perhaps pointless, since this is mostly a proof of existence), but it would be reassuring to see that these details do not substantially affect the results, at least for a couple of examples. Given the simplicity of the analysis, it should be straightforward to try out a few other algorithmic configurations. Importantly, the authors already show that RwM still does not reproduce features of the data when changing the proposal distribution to a heavy-tailed distribution, probably the most relevant control. However, a couple more examples would not hurt.  Clarity: The paper is well-written and easy to follow throughout. Only found a couple of typos: line 111: follows → follow line 131: MCMC, was → MCMC, which was  Originality: This work extends previous computational work in the cognitive sciences that proposed sampling (and specifically MCMC) as an algorithm used by the brain for approximating Bayesian inference. The novelty of this proposal is that it explains a couple of important features of observed data (Lévy flights and 1/f noise) via a MCMC method which was not previously considered as a model for cognition, as far as I know.  Significance: There has been a trend in the past years of describing any chosen aspect of cognition via some specific modern (or less modern) machine learning technique, which is somewhat suspicious given that as humans we always had the tendency to describe the functioning of the brain -- once we started to understand its relevance for cognition -- with comparisons to the latest, fanciest engineering advancement. However, the current proposal is interesting in that some of the algorithmic features (e.g., the existence of different processes/chains, the associated temperatures) might be subject to direct experimental investigation, perhaps making this hypothesis testable beyond some qualitative agreement and proof of concept.  In conclusion, this is a nice conference paper which presents an interesting idea and some preliminary support for it.  After Author Feedback: Thank you for the clarifications. I find positive that the authors are going to share their code (I hope all the code to reproduce the experiments in the paper), and increases my belief in reproducibility of their results.


Neural Proximal Gradient Descent for Compressive Imaging
Recovering high-resolution images from limited sensory data typically leads to a serious ill-posed inverse problem, demanding inversion algorithms that effectively capture the prior information. Learning a good inverse mapping from training data faces severe challenges, including: (i) scarcity of training data; (ii) need for plausible reconstructions that are physically feasible; (iii) need for fast reconstruction, especially in real-time applications. We develop a successful system solving all these challenges, using as basic architecture the recurrent application of proximal gradient algorithm. We learn a proximal map that works well with real images based on residual networks. Contraction of the resulting map is analyzed, and incoherence conditions are investigated that drive the convergence of the iterates. Extensive experiments are carried out under different settings: (a) reconstructing abdominal MRI of pediatric patients from highly undersampled Fourier-space data and (b) superresolving natural face images. Our key findings include: 1. a recurrent ResNet with a single residual block unrolled from an iterative algorithm yields an effective proximal which accurately reveals MR image details. 2. Our architecture significantly outperforms conventional non-recurrent deep ResNets by 2dB SNR; it is also trained much more rapidly. 3. It outperforms state-of-the-art compressed-sensing Wavelet-based methods by 4dB SNR, with 100x speedups in reconstruction time.

While my concerns were given significant attention in the rebuttal, I feel they were not fully addressed. In particular, regarding comparison with deep ADMM-net and LDAMP, the authors argue that these methods need more training data/training time.  However, training time is normally not a big issue (you only train your model once, does it matter if it takes 2 hours or 10?). The *size* of the training data is however important, but no experiments are provided to show superior performance of the proposed method with respect to the the size of training data. This is surprising given that in l. 62. the authors say they use "much less training data" (addressing the challenge of "scarcity of training data" mentioned in l.4 in abstract), without referring back to this claimed contribution anywhere in the paper!  Additionally, for the additional super resolution experiments in Table 1 of the rebuttal, I am left again confused. Instead of reporting on a common SR dataset (like one of the datasets SRResnet[12] reports on), they authors report on CelebA and compare with their own implementation of SRResnet on this dataset. The authors do not show a standard baseline, like Bicubic interpolation, meaning the context for these numbers is missing. When looking up in the literature for results on CelebA 8x upscaling (32x32 -> 128x128), (Wavelet-SRNet: A Wavelet-based CNN for Multi-scale Face Super Resolution, Huang et. al., ICCV 2017) gives wildly different numbers. There we see Bicubic having 29.2dB PSNR (higher than the 28.X reported in the rebuttal!), but SSIM of the s.o.t.a. being 0.9432 (lower than the 95.07 SSIM in the rebuttal...). Either the setting is somehow different, or the experiments are somehow flawed, but it is hard to say given that the authors show no standard baselines!  On the other side, Reviewer 3 gives a quite different perspective on the paper, focusing on the contraction analysis and the accompanying experiments. While I'm not qualified to comment on the correctness/applicability of this part of the paper, I think Reviewer 3 has valid points that such analysis are valuable in general and should be more often done.  ============================================================ The paper proposes an inversion algorithm for ill posed linear measurements of images. That is, from an image x, we are given few (compared to dim(x) )  (noisy) linear measurements from which we want to recover x, relying on the assumption that x lives in some low dimensional manifold. This problem encapsulates for example MRI/CT imaging as well as super-resolution.  The proposed approach is an adaptation of the proximal gradient algorithm for deep networks. While I'm not an expert on proximal learning, my understanding is that the authors take the iterative procedure of proximal learning (eq (2)) and swap out the proximal operator (which has closed form solutions for simple priors but otherwise requires solving a convex optimization problem at each iteration), with a deep network. The intuition is that normally the proximal operator can be viewed as taking a step towards the manifold/prior, which is an operation a DNN could in principle learn.  This means that instead of deriving the operator from the (known) prior, it needs to be learned from data, motivating the loss (P1) in l. 133.   However, this loss is not very clearly explained. Why do we need beta and the (second) measurement consistency term? How does that relate to classical proximal learning? Why not just ask different iterations to also reconstruct the image? How does this term come into the contraction analysis?  When the authors jump to the neural proximal training, they essentially just obtain a constrained recursive image restoration network (in the sense that the relationship between x_t and x_t+1 is the composition of a generic DNN and a fixed linear operator g (that depends on the measurement matrix)).  Previously recursive approaches for image restorationhave been studied, such as ( Deeply-Recursive Convolutional Network for Image Super-Resolution, Kim et al. CVPR 2016 ) and the authors should discuss the differences. Aside from the fixed g, the training losses differ, but again the loss does not have a clear motivation from proximal learning theory as far as I understand.  Experimentally, the paper could also be more convincing.  The MRI experiments, while impressive, seem to be done on a private dataset with limited comparisons. Why can't the authors e.g. compare with (Deep ADMM-Net for Compressive Sensing MRI, Sun et al, NIPS 2016) or  ( Learned D-AMP: Principled Neural Network Based Compressive Image Recovery, Metzler et al, NIPS 2017)?  The Super-Resolution experiments (a field I know well) are extremely limited. The authors claim they only do them to analyse the contraction properties, but do not explain why they need to switch to SR and cannot do this for the MRI experiments. For me, the experiments rather rather raise the question on why the method works so poorly for SR, since this is a problem that falls within the problem formulation (linear measurement matrix). Note I am not saying the authors need to be state of the art in SR, but at least they should report results in a standard manner (e.g. on standard datasets such as Set14, see e.g. Image Super-Resolution Using Dense Skip Connections, Tong et al, ICCV 2017 for a recent s.o.t.a. SR paper).  Overall, I think the goal of the paper is worthy (generalizing proximal algorithms to deep net), and the paper is well written in general. My main concern is the execution, both in terms of how the proximal method actually motivates the training objective and the overall experimental evaluation.   Minor Comments: l 105: the definition of g is just the argument to the proximal operator in eq (2). I would either define g earlier or refer to (2) in l105. l 129: Shouldn't it be \hat{x}_i = x_i^T instead of x_T^i?


One-Shot Unsupervised Cross Domain Translation
Given a single image x from domain A and a set of images from domain B, our task is to generate the analogous of x in B. We argue that this task could be a key AI capability that underlines the ability of cognitive agents to act in the world and present empirical evidence that the existing unsupervised domain translation methods fail on this task. Our method follows a two step process. First, a variational autoencoder for domain B is trained. Then, given the new sample x, we create a variational autoencoder for domain A by adapting the layers that are close to the image in order to directly fit x, and only indirectly adapt the other layers. Our experiments indicate that the new method does as well, when trained on one sample x, as the existing domain transfer methods, when these enjoy a multitude of training samples from domain A. Our code is made publicly available at https://github.com/sagiebenaim/OneShotTranslation.
latent space vectors. In order to encourage generality, we further augment B with samples produced by a slight rotation and with a random horizontal translation.
In the second phase, the variational autoencoder is cloned to create two copies that share the top layers of the encoders and the bottom layers of the decoders, one for the samples in B and one for the sample x in A. The autoencoders are trained with reconstruction losses as well as with a single-sample one-way circularity loss. The samples from domain B continue to train its own copy as in the first step, updating both the shared and the unshared layers. The gradient from sample x updates only the unshared layers and not the shared layers. This way, the autoencoder of B is adjusted by x through the loss incurred on unshared layers for domain B by the circularity loss, and through subsequent adaptation of the shared layers to fit the samples of B. This allows the shared layers to gradually adapt to the new sample x, but prevents overfitting on this single sample. Augmentation is applied, as before, to B and also to x for added stability.
We perform a wide variety of experiments and demonstrate that OST outperforms the existing algorithms in the low-shot scenario. On most datasets the method also presents a comparable accuracy with a single training example to the accuracy obtained by the other methods for the entire set of domain A images. This success sheds new light on the potential mechanisms that underlie unsupervised domain translation, since in the one-shot case, constraints on the inter-sample correlations in domain A do not apply.

In overall, I think this paper proposes a well-designed two steps learning pipeline for one-shot unsupervised image translation. But the explanations about selective backpropagation in the rebuttal are still not clear to me. According to Eq.8-14, it seems that G^S and E^S are not updated in phase II. But according to the Tab. 1 and the rebuttal, they seem to be selectively updated. I strongly suggest the authors to explain the details and motivation in the method part if this paper is accepted. I keep my initial rating. =========================================== This paper proposes a two steps method for one-shot unsupervised image-to-image translation task. This method can help to enable the one-shot unsupervised image translation and also improve the performance continuously given more training samples.  Pros: * The proposed idea is interesting that designing a two steps learning pipeline based on the shared latent space assumption which is implemented with the simple weight sharing technique. * The writing is easy to follow, and empirical experiments show that it is able to translate the image in one-shot setting.  Cons: * If I understand correctly, the shared layers of G^S and E^S will only be updated with loss L^I in phase two of training. The explanation for such selective backpropagation in line 146-148 is not so convincing to me.  ** I would like the authors to explain the function and motivation for such selective backpropagation in more details, since it is the key for the one-shot unsupervised image-to-image translation. ** Does the training of G^S and E^S aim at keeping the shared latent constraint?  ** Why only use one-way cycle loss? I would like the authors to explain the reason of not using two-way cycle loss and show the comparison.  * In the results of style transfer and drawing tasks, the authors only show the OST 1-shot results. I wonder how the proposed method will perform on these tasks with all-shot setting, since for MNIST to SVHN Translation OST is able to perform better than CycleGAN and UNIT when more samples are presented.


Learning Latent Subspaces in Variational Autoencoders
Variational autoencoders (VAEs) [10,20] are widely used deep generative models capable of learning unsupervised latent representations of data. Such representations are often difficult to interpret or control. We consider the problem of unsupervised learning of features correlated to specific labels in a dataset. We propose a VAE-based generative model which we show is capable of extracting features correlated to binary labels in the data and structuring it in a latent subspace which is easy to interpret. Our model, the Conditional Subspace VAE (CSVAE), uses mutual information minimization to learn a low-dimensional latent subspace associated with each label that can easily be inspected and independently manipulated. We demonstrate the utility of the learned representations for attribute manipulation tasks on both the Toronto Face [23] and CelebA [15]  datasets.

Updated (due to rebuttal & discussion w/ R2):  The authors reiterate in their rebuttal their core contributions of "extracting information beyond binary labels" and "attribute manipulation from a single image", together with the promise to clarify it in the paper. The contributions are relevant to the community, since this form of hierarchical disentangling seems novel.  That said, there is some degree of similarity of the proposed variational approach to IGN (Deep Convolutional Inverse Graphics Network https://arxiv.org/abs/1503.03167). IGN is cited, but not discussed in detail, and an empirical comparison is not provided, despite being applicable to the current setting as well.  Nevertheless, since the selling point of the paper seems to be the ability to discover sub-categories from only category labels, which is not addressed in IGN and is an interesting empirical find, I increased my score to be marginally above the acceptance threshold.  ---  The paper proposes an approach for learning variational auto-encoders for labeled data in which the latent space consists of two parts: one that captures the labels information and one that captures the remaining information for producing the data. Along with the variational lower-bound corresponding to the introduced latent variables, the proposed training objective additionally penalizes the mutual information between the latent variable corresponding to the remaining data and the labels. The approach is illustrated on a toy example and two image datasets, the Toronto Faces dataset and CelebA.  I think the approach is described clearly, and several experiments, qualitative and quantitative, are presented. That said, since there are many works on the topic, some seemingly quite similar in the use of variational auto-encoders with various regularizers for disentangling latent representations, I worry that the proposed approach only incrementally contributes to the topic. See below for specific questions on this matter.  Comments: * The work on building fair representations (where the learned representations are encouraged to be independent of "sensitive" factors) seems relevant and not discussed. Could you highlight the distinction? For instance, The Variational Fair Autoencoder [1] seems relevant, where the additional "fairness" regularizer uses maximum mean discrepancy rather than mutual information. From this perspective, I believe that the contribution of the current paper is the use of the mutual information regularizer, and a discussion on the benefits of one metric versus the other would be useful. From a quick search, the idea of using mutual information to "obfuscate" sensitive information is also mentioned in [3], though in a somewhat different setting. * In the related work section, I wish the authors provided more detail of the proposed approach with respect to the cited works (e.g. w.r.t. [21-23] and w.r.t. [3]) so that we can understand which specific contributions are being made and how to evaluate their merits. For instance, the fact that previous work used GANs rather than VAEs (lines 61-62) is a pretty weak point in favor of the paper, unless properties of VAEs are shown to be necessary for solving the task, or the method shown to empirically perform much better than the GAN counterparts.  Experiments: * I like the toy example, it's illustrative and a good sanity check for the method. * None of the presented applications is focused on the specific contributions of this paper, e.g. the additional mutual information regularizer. Instead, the experiments are quite generic for attribute manipulation/style transfer which have been presented in many works recently, e.g. the ones mentioned in the related work. From this perspective, the empirical contribution does not stand out.  [1] https://arxiv.org/abs/1511.00830 ICLR 2016 [2] https://www.cs.toronto.edu/~toni/Papers/icml-final.pdf ICML 2013  Summary: The submission is ok, but, unless the contributions are highlighted more clearly, it only incrementally contributes to the literature, hence the slightly negative score.