We push on the boundaries of our knowledge about dropout by showing theoretically that dropout training can be understood as performing MAP estimation concurrently for an entire family of conditional models whose objectives are themselves lower bounded by the usual dropout objective. This discovery allows us to pick any model from this family after training, which leads to a substantial improvement on regularisation-heavy language modelling. The family includes models that compute a power mean over the sampled dropout masks, and their less stochastic subvariants with tighter and higher lower bounds than the fully stochastic dropout objective. The deterministic subvariant's bound is equal to its objective, and the highest amongst these models. It also exhibits the best model fit in our experiments. Together, these results suggest that the predominant view of deterministic dropout as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective.

Summary:
The paper aims to develop a more principled framework for choosing between different inference procedures in neural network models employing dropout as a stochastic regularizer. In particular, they posit a family of conditional models and show that the learning objectives of all these models are lower bounded by the usual dropout training objective with L2 regularization on weights. They proceed to show empirically that the deterministic inference procedure (multiplying the node's output by the droput rate) achieves the tightest lower bound. From this observation the authors conclude that deterministic inference should be seen as the best available approximation to the true dropout objective rather than an approximation to Monte Carlo averaging.
Strengths: The paper builds on recent works viewing dropout as a Bayesian approximation to the predictive posterior distribution. Introducing a conditional model and showing that the dropout objective is akin to MAP estimation of the parameters of this model is interesting. The result that dropout simultaneously optimizes a lower bound to an entire family of conditional distributions is novel.
Weaknesses:
The writing is often not clear, ambiguous or misleading and needs improvement. For instance:
In Section 2.1, comments on weaknesses of variational dropout seems out of place. It should either be ommitted or shifted to the previous section where variational dropout is introduced.
In Section 2.1, "Consider a conditional model p(Y |X, Θ) as a crippled generative model with p(X) constant, X and Θ independent." Assuming p(X), the input features, to be constant is a very strong assumption. The variational lower bound is still true if p(X) is assumed arbitrary independent of Θ.
In Section 3.3, last paragraph, "Suppose we pick a base model from the power mean family and have a continuum of subvariants with gradually reduced variance in their predictions but the same expectation." It is not clear what the authors mean by continuum of subvariants? Is it the dropout rate?
Several statements are made without any citations or explanations.
In Section 3. "While it is easy to argue in general that objectives of more than one model may share any given lower bound" How? More explanation needed.
In Section 3.1, "Notice how with SGD and multiple epochs, for each data point several dropout masks are encountered, and the approximating quantity is the geometric mean of the predicted probabilities". Citation needed.
In Section 3.2, "because M_α is monotonically increasing in α". Proof (in appendix) or citation needed.
In Section 5, "The construction of a conditional model family with a common lower bound on their objectives is applicable to other latent variable models with similar structure and inference method." What general structure and inference method are the authors referring to?
Technical Concerns:
Section 3.3, last paragraph. The entire paragraph is extremely convoluted. It is not clear how one achieves deterministic dropout inference by reducing variance of the prediction y keeping its expectation constant.
Section 3.3, "A similar argument based ... shows that Z monotonically increases... " How? The authors should deliberate more on this statement since, the Z term is also important in the difference between the true objective of each conditional model and the droput training objective.
In Section 5, "The gains reported in those works might be explained by reducing the bias of deterministic evaluation and also by encouraging small variance in the predictions and thus getting tighter bounds." Isn't this contradictory to Section 3.3, where from Eq. 10 and Eq. 11? If the bias is reduced and variance increases, according to Eq. 10, the lower bound would become looser.
How is Fig. 1a and 1b computed?
Conclusions drawn from experiments not convincing.
Section 3.4, last line. "Having trained a model with dropout, the best ﬁt is achieved by the deterministic model with no dropout. This result isolates the regularisation effects from the biases of the lower bound and the dropout family." How does this isolate the regularization effects? What biases of the lower bound? Do you mean the difference between the model's true objective and the dropout training objective?
Table 4 indicates that not only AMC, also power with alpha=0.5 is better than deterministic in some cases. Did the authors try other values of alphas? Deterministic seems to be good just for MNIST. This strongly refutes the most important claim of this paper, written in the abstract, "Together, these results suggest that the predominant view of deterministic dropout as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective."
Changing the dropout multiplier and adjusting the softmax temperature of the network output layer, to achieve comparable performance to AMC on several datasets seems to support the existing hypothesis that deterministic dropout is a good approximation to MC average, and not the other way around!
Summary:
The paper introduces some interesting ideas about dropout but suffers from bad writing and presentation of results. One of the most important claims made in this paper, "dropout trains a deterministic model ﬁrst and foremost and a continuum of stochastic ones to various extents", is not well-motivated theoretically in Section 3.3. Consequently, this seems to be purely an empirical observation, which is contradicted by further experiments on linguistic datasets. Several conclusions made from experiments seem adhoc.


The classical Universal Approximation Theorem certifies that the universal approximation property holds for the class of neural networks of arbitrary width. Here we consider the natural 'dual' theorem for width-bounded networks of arbitrary depth. Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonaffine continuous function, with a continuous nonzero derivative at some point. Then we show that the class of neural networks of arbitrary depth, width n + m + 2, and activation function ρ, exhibits the universal approximation property with respect to the uniform norm on compact subsets of R n . This covers every activation function possible to use in practice; in particular this includes polynomial activation functions, making this genuinely different to the classical case. We go on to consider extensions of this result. First we show an analogous result for a certain class of nowhere differentiable activation functions. Second we establish an analogous result for noncompact domains, by showing that deep narrow networks with the ReLU activation function exhibit the universal approximation property with respect to the p-norm on R n . Finally we show that width of only n + m + 1 suffices for 'most' activation functions.

This paper proves universal approximation theorems for fully connected networks with fixed width and unbounded depth. Unlike recent results focusing on ReLU networks approximating scalar valued target functions, this paper provides universal approximation theorems for a wide range of activation functions and vector valued target functions.
This paper provides a number of universal approximation theorems on different activation functions, but the central result can be stated as follows:
Given input dimension n and output dimension m and activation \rho, assume \rho is a continuous function and there exists \alpha \in R such that \rho is continuously differentiable at \alpha and the derivative \rho’ is nonzero at \alpha. Then, a fully connected network with layer width n+m+2 and unbounded depth can approximate any continuous function on a compact domain to arbitrary sup-norm accuracy.
I would like to vote for acceptance of this paper because the authors develop nontrivial techniques that extend existing universal approximation results on width-bounded ReLU networks to essentially “all” other activation functions. I think this paper is well-written and well-organized; it gives a good overview of the existing results that contextualizes this paper well, briefly summarizes the main results, and then reveals the details of construction. I haven’t checked the full details of the proofs in the appendix, but as far as I can tell they look correct.
While I think that Section 4 reveals the proof techniques reasonably well, I believe that proofs of Propositions 4.3 and 4.7 should be covered/sketched in greater detail in the main text. For example, Theorem 4.4 builds upon Proposition 4.3 by approximating identity function locally using \rho. The main text only reveals approximation of identity function by \rho and defers the whole proof of Proposition 4.3 to the appendix. I think adding proof sketches for the propositions will be more helpful to the readers, and to this end, cutting down some text in the recurring paragraph “uniform continuity preserves uniform convergence…” should be helpful.
According to Remark 4.9, it seems that the proof strategy for polynomials can also be applied to nonpolynomials. This motivates a natural question: why can’t you apply the proof strategy for nonpolynomials to polynomials? I believe you can’t because Proposition 4.3 relies on the existing universal approximation results which requires \rho to be nonpolynomial (which is not mentioned in the main text). In my opinion, adding some comments on “why nonpoly techniques can’t be applied to poly activations” would help readers better understand the proof techniques.
Another question just out of curiosity: can you prove tightness of your construction, e.g., show that a square model with width n+m is NOT a universal approximator?


Designing efficient algorithms for combinatorial optimization appears ubiquitously in various scientific fields. Recently, deep reinforcement learning (DRL) frameworks have gained considerable attention as a new approach: they can automatically train a good solver while relying less on sophisticated domain knowledge of the target problem. However, the number of stages (until reaching the final solution) required by existing DRL solvers is proportional to the size of the input graph, which hurts their scalability to large-scale instances. In this paper, we seek to resolve this issue by proposing a novel design of DRL's policy, coined auto-deferring policy (AutoDP), automatically stretching or shrinking its decision process. Specifically, it decides whether to finalize the value of each vertex at the current stage or defer to determine it at later stages. We apply the proposed AutoDP framework to the maximum independent set (MIS) problem and its variants under various scenarios. Our experimental results demonstrate significant improvement of AutoDP over the current state-of-the-art DRL scheme in terms of computational efficiency and approximation quality. The reported performance of our generic DRL scheme is also comparable with that of the existing solvers for MIS, e.g., AutoDP outperforms them for the Barabási-Albert graph with two million vertices.

The paper proposes a Deep RL approach called Auto-Deferring Policy (ADP) to learning a policy for constructing solutions for the Maximum Independent Set (MIS) problem. Rather than constructing a solution one variable per episode step, the policy can make decisions about multiple variables per step, as well as defer decisions to later steps. At each step the MIS constraints of a valid solution are checked and decisions that violate the constraints are reverted, and any additional decisions that are automatically implied by the decisions so far and the MIS constraints are taken. The policy and value function are parameterized as a graph convolutional network to make use of the graph structure of the problem. A diversity regularizer that encourages the policy to generate diverse final solutions is used for training. Results show that the approach is able to match the objective value of the state-of-the-art solvers on several datasets, and is able to outperform S2V-DQN on several datasets when neither approach is trained on them.
Pros:
- Extensive experiments have been done on several datasets, both synthetic and real. The ablation and generalization experiments are particularly valuable for getting insight into the performance of the algorithm.
- Comparison to CPLEX and ReduMIS strengthens the results.
- The paper is reasonably well-written and easy to follow. Figures 2 and 3 are especially useful for quickly understanding key ideas.
Cons:
- The update and clean-up phases inject significant domain knowledge about MIS into the policy. At a high level the idea is similar to unit propagation in Boolean Satisfiability (SAT) solvers or domain propagation in Mixed Integer Programming (MIP) solvers, i.e., the hard constraints of the problem can be used to make decisions in the search for a solution. Both SAT and MIP solvers have been shown to rely on it to significantly improve their search. Improvements over S2V-DQN could be due to this extra built-in domain knowledge, rather than any improvements related to learning. For a fairer comparison, either this knowledge should be removed from ADP as an additional ablation study, or it should somehow be given to S2V-DQN as well. Without such a comparison, it is not clear that the improvements are really due to the new ideas like auto-deferring, diversity regularizer, etc.
- A comparison to Li, Chen, and Koltun, NeurIPS’18 is needed since they have shown strong results for MIS and they have made the code available online. Although their approach is supervised learning, it would still be useful to assess an RL approach’s performance relative to SL.
Additional comments:
- It is not clear what encourages the policy to defer decisions, rather than making all the {0,1} decisions for all variables in one step or a small number of steps. Does this behaviour of the policy arise naturally via training, or does some regularization need to be applied to ensure that the policy doesn’t prematurely fix all variables? For example, as a way to avoid/reduce variance in the reward signal, the policy may learn to fix all variables in just one step -- does this issue arise?
- The abstract says "The reported performance of our generic DRL scheme is also comparable with that of the state-of-the-art solvers specialized for MIS, e.g., ADP outperforms them for some graphs with millions of vertices." Can you please point out the specific results in the paper that this sentence is referring to?
- Comparison to CPLEX:  The paper mentions that “we observe that our algorithm outperforms the CPLEX solver on ER-(100, 200) and ER-(400, 500) datasets, while consuming a smaller amount of time”, and “It is remarkable to observe that ADP outperforms the CPLEX solver on both datasets under reasonably limited time.” Note that CPLEX not only optimizes the objective, but also proves a bound on the objective, while ADP only does the former. So this is not a fair comparison. A fairer comparison would be to set CPLEX hyperparameters to give higher priority to optimizing the objective and then measure the time CPLEX took to first reach a particular objective value rather than the time to “solve” an instance. It can be the case that a given objective value is achieved quickly but then proving a bound requires much longer. Also there is no mention of the optimality gap used as a stopping criterion for CPLEX.
- One potential advantage of making one decision per step is that the credit assignment problem may be simpler compared to making many decisions simultaneously. It would be insightful to explore this tradeoff in more detail.
- The acronym ADP is somewhat well-known in the optimization community as Approximate Dynamic Programming (http://adp.princeton.edu/), so it would be helpful to use a different one.


In this paper, a new population-guided parallel learning scheme is proposed to enhance the performance of off-policy reinforcement learning (RL). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The key point is that the information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search region by the multiple learners. The guidance by the previous best policy and the enlarged range enable faster and better policy search. Monotone improvement of the expected cumulative return by the proposed scheme is proved theoretically. Working algorithms are constructed by applying the proposed scheme to the twin delayed deep deterministic (TD3) policy gradient algorithm. Numerical results show that the constructed algorithm outperforms most of the current state-of-the-art RL algorithms, and the gain is significant in the case of sparse reward environment.

The authors propose another method of doing population-based training of RL policies. During the training process, there are N workers running in N copies of the environment, each with different parameter settings for the policies and value networks. Each worker pushes data to a shared replay buffer of experience. The paper claims that a natural approach is to have a chief job periodically poll for the best worker, then replace the weights of each worker with the best one. Whenever this occurs, this reduces the diversity within the population.
In its place, the authors propose a soft-update in the chief. At every merging cycle, the chief queries which worker performs best. If that worker is worker B, it emits pi_B's parameters to each of the other workers. Instead of replacing the parameters exactly, worker i's loss is then augmented by beta * D(pi_i, pi_B), where D is some distance measure that is measured over states sampled from the replay buffer. The "soft" update encourages individual workers to match pi_B without directly replacing their parameters, which maintains diversity in the population. In this work, pi is always represented by a deterministic policy and D is the mean-squared-error in action space (this is argued as equivalent to the KL divergence between the two policies if the policies were represented by Gaussian with the same, constant standard deviation). The beta parameter is updated online using heuristics based on how D(pi_i, pi_B) compares to D(pi_i, old_pi_i). Using TD3 as a base algorithm, the population-based version performs better, and there are ablations for various parts of the population algorithm.
I thought this paper was interesting, but thought it was strange that there were very few comparisons to other population / ensemble-based training methods. In particular they mention the copying problem as a downside of population-based training (PBT), but do not compare against PBT at all. Additionally, my understanding of PBT is that when they replace bad agents with the best agent, they only replace the worst performing agents (not all of them), and they additionally add some random perturbations to their hyperparameter settings. This goes counter to the claim that they collapse the population to a single point- by my reading the exploration step avoids this collapse.
An experiment I'd like to see is trying PBT, where different workers do in fact use different hyperparameters. My understanding is that in P3S-TD3 there is a single hyperparameter setting shared across all workers (plus some hyperparameters deciding the soft update).
I'd also like to see ablations for the Resetting variant (Re-TD3), where only the bottom half or 2/3rds of the workers are reset. This would give empirical evidence for the "population collapse" intuition - we should expect to see some improvements if we avoid totally collapsing the population, while still copying enough to partially exploit the current best setting.
Many inequalities in the paper are argued by compare the expectation of negative Q of one policy to the negative Q of another - I believe the derivations would be much easier to follow if the authors simply multiplied all sides by -1 and adjusted inequalities accordingly. It is much easier to think about Q-value-1 > Q-value-2 rather than -Q-value-1 < -Q-value-2 when trying to interpret what the equation is saying.
For related work, papers on evolutionary strategies and the various self-play-in-a-population papers seem relevant, since these often take the form of having each worker i do a different perturbation that is later merged by a chief.
In Figure 4 it feels weird that results are the regular Mujoco envs for 2 problems and the delayed envs for the other 2 problems. When looking at the appendix, it's rather clearly cherry picked to show the best results in favor of PS3-TD3. I would prefer the Delayed MuJoCo experiments be in a separate figure, or to include the TD3/SAC/ACKTR/PPO/etc. results for the delayed envs as well (these don't appear to be in the appendix)
On the theoretical results: the 1st assumption seems very strong. The first assumption argues that pi_B is always 1-step better than pi_old for every state. That assumption already takes you very far towards arguing "updating pi_old to pi_B is good". The 2nd assumption is more reasonable but I'm confused how rho and d play into the theoretical results. Do they play any role in how much the policy is expected to improve, or do the constants just need to exist?
The last comment on the theory side is that I still don't understand the intuition for why we want to learn beta such that
KL(pi_new || pi_b) = max {rho * KL_max(pi_new || pi_old), d}
In the practical algorithm, beta is updated online to increase / decrease the importance of the "match pi_B" term if the ratio between the two strays too far from 1 (with the threshold set to [1/1.5, 1.5] in a manner similar to PPO's approach). But why should it be important for the two values to be close to one another? Let me write out the derivation continuing from Eqn (57) in the appendix.
With a substitution that doesn't use (c) to drop the beta * (KL - KL) term, we get
E_{pi_b}[-Q_new] >= E_{pi_new}[-Q_new] + beta * (KL - KL)
-->
E_{pi_new}[Q_new] >= E_{pi_b}[Q_new] + beta * (KL - KL)
Then, in Theorem 1, we recursively apply this inequality, accumulating a number of beta * (KL - KL) terms. In the end we get
Q_new >= (discounted sum rewards from pi_b) + (discounted sum of beta * (KL - KL) with expectation over states from pi_b)
= Q_pi_b + (sum of beta *(KL - KL) terms)
By my reading, shouldn't this mean we want KL(pi_new || pi_b) - max {rho * KL_max(pi_new || pi_old), d} to be as large as possible, rather than 0? The more positive this term is, the more improvement we get between Q_new and Q_pi_b.
--------------------------
Overall, this feels like a good paper, but I'm not too familiar with prior empirical results for population-based RL methods. The ablations suggested that pretty much any reasonable population-based method outperformed using a single worker, and because of this it seems especially important to have ablations to other population-based prior work, rather than just variants of its own method.
I would be okay with this paper as-is despite some of its flaws, but think it could be better pending rebuttal.
Edit: I have read the author reply and other reviews. I do not plan to change my rating but do think the paper is improved by the added baselines and better explanation of what the beta adaptation rule is doing. I would ask the authors to make sure this description is as clear as possible, as the argued improvement gap seems central to the work.


The vulnerability to slight input perturbations is a worrying yet intriguing property of deep neural networks (DNNs). Though some efforts have been devoted to investigating the reason behind such adversarial behavior, the relation between standard accuracy and adversarial behavior of DNNs is still little understood. In this work, we reveal such relation by first introducing a metric characterizing the standard performance of DNNs. Then we theoretically show this metric can be disentangled into an information-theoretic non-robust component that is related to adversarial behavior, and a robust component. Then, we show by experiments that DNNs under standard training rely heavily on optimizing the non-robust component in achieving decent performance. We also demonstrate current state-ofthe-art adversarial training algorithms indeed try to robustify DNNs by preventing them from using the non-robust component to distinguish samples from different categories. Based on our findings, we take a step forward and point out the possible direction of simultaneously achieving decent standard generalization and adversarial robustness. It is hoped that our theory can further inspire the community to make more interesting discoveries about the relation between standard accuracy and adversarial robustness of DNNs.

The authors propose a new performance metric, CCKL, as an alternative to standard cross-entropy. At a high level the goal of this objective is to ensure that the classifier is confidently separating samples from different classes. They decompose this objective into its Taylor expansion terms and argue that, since adversarial vulnerability is a local property, it only depends on the first of these terms. They finally propose training models of higher capacity as a way of relying more on higher-order terms of the objective and hence improving adversarial robustness.
I found the core idea potentially interesting. It is an attempt at rigorously studying the tension between local stability (necessary for robustness) and the fact that different classes should be confidently separated. However, I have several concerns regarding the theoretical derivation and experimental evaluation.
First of all, I did not find the proposed CCKL objective well motivated. Theorem 3.1 only shows that standard training improves a _lower bound_ on the proposed objective. However, this only implies that this objective is being optimized as a byproduct of standard cross-entropy optimization. It is unclear for instance if this is a good performance indicator or whether training with this objective will even lead to a good classifier (for instance the classifier could learn a permutation of the labels instead). Therefore, I do not find the theoretical and experimental evidence in favor of studying this proposed objective convincing.
Moreover, the experimental evidence presented is quite weak. While the authors compare various quantities related to standard and robust models, they do not perform the ablations necessary to understand whether these properties are causally related to robustness of simply a byproduct of the specific training scheme.
Finally, the conclusion that larger capacity can improve both model accuracy and robustness has already been made in prior work (Kurakin et al. 2016, Madry et al. 2017). The intuition that added capacity helps by allowing the model to leverage higher order terms of the loss is interesting but a rather incremental contribution.
Overall, while the ideas presented in the paper could be of interest, they are not rigorously established and rather serve as high-level intuition at this moment. Moreover, the writing of the paper could be significantly improved. It took me quite some time to distill the above intuition despite the fact that it is a rather elementary consequence of the Taylor expansion (I am not sure what the discussion about information geometry adds to this work). I would thus encourage the authors to continue working on these ideas and their presentation. However, for now, I will have to recommend rejection.


We introduce a novel architecture that integrates a large addressable memory space into the core functionality of a deep neural network. Our design distributes both memory addressing operations and storage capacity over many network layers. Distinct from strategies that connect neural networks to external memory banks, our approach co-locates memory with computation throughout the network structure. Mirroring recent architectural innovations in convolutional networks, we organize memory into a multiresolution hierarchy, whose internal connectivity enables learning of dynamic information routing strategies and data-dependent read/write operations. This multigrid spatial layout permits parameter-efficient scaling of memory size, allowing us to experiment with memories substantially larger than those in prior work. We demonstrate this capability on synthetic exploration and mapping tasks, where the network is able to self-organize and retain long-term memory for trajectories of thousands of time steps. On tasks decoupled from any notion of spatial geometry, such as sorting or associative recall, our design functions as a truly generic memory and yields results competitive with those of the recently proposed Differentiable Neural Computer (Graves et al., 2016).

The paper proposes a multigrid memory architecture by introducing multigrid CNN [1] into convolutional LSTM network [2]. The method extends the convolutional LSTM with bigger memory capacity in forms of multigrid CNNs. Some specific designs such as multiple threads and encoder-decoder are also proposed. The model is validated with synthetic tasks such as spatial mapping, associative recall and priority sort.
Pros:
* The method is well-motivated. Utilizing multigrid hierarchy enables convolutional LSTM to operate across scale space and thus may achieve richer representation for the hidden state memory.
* The experiments are well designed (especially RL tasks), demonstrating the advantage of the proposed model.
Cons:
* The proposed model is not presented clearly. The paper does not show details on how [1] and [2] are integrated, which requires the reader to refer back to the old works and make inference on the integration. Besides graphic illustration, the authors should include a brief review on [1, 2] and introduce some basic formulas describing the combination between the two.
* Section 3.2 is supposed to contain the most important design considerations, but details seem missing. For example, what does the Writer do to the memory?
* The contribution is rather incremental. Without detailed description, it seems that the proposed model is a straightforward replacement of the vanilla CNN with another CNN (multigrid CNN) in the convolutional LSTM architecture.
* The experiments are not very satisfactory for the “generality” claim that the paper makes.
Questions and concerns:
* Could you explain the term “addressable memory space”? It seems that your network’s memory comes from the internal states of LSTM. How is the memory addressable?
* The analysis on information routing seems interesting. However, how does it relate to the memorization capacity? Is there any guarantee that the information from source grid is preserved in higher levels/layers? How does it differ from using vanilla multiple-layer neural networks?
* As DNC is not originally designed for image inputs, how did you feed the images to DNC? Did you tune DNC carefully by adjusting the number of elements per memory slot? Also, DNC seems not a really strong baseline. Other solutions to increase memory capacity of MANNs exist [3, 4]
* For algorithmic tasks, why don’t you include ConvLSTM as a baseline? Also, NTM maybe a better baseline than DNC for these tasks.
* What is the model size and computational complexity compared to other MANNs?
* The model is naturally fit for visual inputs. Is there any advantage when applying it to other sequential data (NLP, time-series)?
Reference
[1] Ke, Tsung-Wei, Michael Maire, and Stella X. Yu. "Multigrid neural architectures." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6665-6673. 2017.
[2] Xingjian, S. H. I., Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. "Convolutional LSTM network: A machine learning approach for precipitation nowcasting." In Advances in neural information processing systems, pp. 802-810. 2015.
[3] Rae, Jack, Jonathan J. Hunt, Ivo Danihelka, Timothy Harley, Andrew W. Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. "Scaling memory-augmented neural networks with sparse reads and writes." In Advances in Neural Information Processing Systems, pp. 3621-3629. 2016.
[4] Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization. In International Conference on Learning Representations, 2019. URL
https://openreview.net/forum?id=r1xlvi0qYm.


Convex relaxations are effective for training and certifying neural networks against norm-bounded adversarial attacks, but they leave a large gap between certifiable and empirical robustness. In principle, convex relaxation can provide tight bounds if the solution to the relaxed problem is feasible for the original non-convex problem. Therefore, we propose two regularizers that can be used to train neural networks that yield tighter convex relaxation bounds for robustness. In all of our experiments, the proposed regularizers result in higher certified accuracy than non-regularized baselines.

Summary:
The aim of the paper is to improve verified training. One of the problem with verified training is the looseness of the bounds employed so the authors suggest incorporating a measure of that looseness into the training loss. It is based on a reformulation of the relaxation of Weng et al.
Comments:
Page 2: "it can certify a broader class of adversaries that IBP cannot certify, like the `2 adversaries.". You can definitely use IBP to very properties against L2-adversaries. It is simply a matter of changing the way the bound is propagated through the first layer.
Page 3: It's a bit pedantic, but the convex relation of Ehlers (middle of figure 1) is not the optimal convex relaxation. It is optimal only if you assume that all ReLU non linearities are relaxed independently. See the work by Anderson et al. for some examples
Page 5, section 4: "We investigate the gap between the optimal convex relaxationin Eq. O" There is a bit of confusion in this section. Eq O is not the optimal convex relaxation, it's the hard non-convex problem.
Section 4.1 bothers me. Equation C is the relaxed version of equation O, so they are only going to be equal if there is essentially no relaxation going on. Saying that it's possible to check whether the equivalence exists is a bit weird. The only case where this can happen is if all the terms in the sum over I_i are zero, which is essentially going to mean that no ReLU is ambiguous. (or if the c W are all positives, but that would be problematic during the optimization of the intermediate bounds given that c would make them both signs then)
Page 5, section 4.2: The authors suggest minimizing d, the gap between the value of the bound obtained, and the value of forwarding the solution of the relaxation through the actual network. Essentially, this would amount to maximizing the lower bound (which all verified training already does), at the same time as minimizing the value of the margin (p_O) on a point of the neighborhood for which we want robustness (x + delta_0). Minimizing the value of the margin is the opposite of what we would want to do, so I'm not surprised by the observation of the author that this doesn't work well.
The conclusion of the section that d can not be optimized to 0 also seems quite obvious if you think about what the problem is.
Section 4.3:
"the optimal solution of C can only be on the boundary of the feasible set." -> There is a subtlety here that I think the authors don't address. The three points they identify are the only feasible optimal solutions for solving a linear program over the feasible domain given by the relaxation of one ReLU but, when solving over the whole of C, the solution needs to be on the boundary of the feasible domain of C, which is larger than those three points.
The whole section is quite convoluted and makes very strong assumption. For Proposition 1, the condition x \in S(\delta) means that all the intermediate bound in the network must have been tight (so that the actual forwarding of an x can match the upper or lower bound used in the relaxation), and that the optimal solution of the relaxation requires all intermediate points to be at either at their maximum or their minimum. The only case I can visualise for this is essentially once again the case where there are no ambiguous ReLU and the full thing is linear.
Regarding the experiments section, it would be benefical to include in table 1 the results of Gowal et al. (On the Effectiveness of Interval Bound propagation for Training Verifiably Robust Models) for better context. The paper is already cited so it should have been possible to include those numbers, which are often better than the ones reported here.
The comparison is included in table 2, when the baseline is beaten, but this is with using the training method of CROWN-IBP and it seems like most of the improvements is due to CROWN-IBP.
Typos/minor details:
Page 2: " In addition, a bound based on semi-definite programming (SDP) relaxation was developed and minimized as the objective Raghunathan et al. (2018). (Wong & Kolter, 2017) presents an upper bound" -> citation format
Page 8: "CORWN-IBP "
Opinion:
I think that the analysis section is pretty confusing and needs to be re-thought. It provides a lot of complex discussion of when the relaxation will be exact, without really identifying that it will be when you have very few ambiguous ReLU. I think that there might be a few parallels to identify between the regularizer proposed and the ReLU stability one of Xiao et al. (ICLR2019) from that aspect. The experimental results are not entirely convicing due to the lack of certain baselines.


Recently, deep learning has made extraordinary achievements in text classification. However, most of present models, especially convolutional neural network (CNN), do not extract long-range associations, global representations, and hierarchical features well due to their relatively shallow and simple structures. This causes a negative effect on text classification. Moreover, we find that there are many express methods of texts. It is appropriate to design the multi-input model to improve the classification effect. But most of models of text classification only use words or characters and do not use the multi-input model. Inspired by the above points and Densenet (Huang et al. (2017)), we propose a new text classification model, which uses words, characters, and labels as input. The model, which is a deep CNN with a novel attention mechanism, can effectively leverage the input information and solve the above issues of the shallow model. We conduct experiments on six large text classification datasets. Our model achieves the state of the art results on all datasets compared to multiple baseline models.

Summary
This paper introduces a new model architecture for doing text classification. The main contribution is proposing a deeper CNN approach, using both word and character embeddings (as well as label embeddings with attention). The paper claims improved performance over baselines.
Decision
I reject the paper for 3 main reasons:
1) Very misleading claims regarding establishing a new state of the art. The baselines used for comparison don't include any of the best existing published results.
2) Lack of positioning within the literature. In particular, no mention nor discussion of Transformers (self-attention) networks, including BERT and XLNet approaches, which are the state of the art in text classification.
3) Lack of justification/explanation for the proposed architecture. One key argument made is that current models are shallow, but it appears that only CNN models are considered for that comparison. More discussion is needed to understand why the new aspects of the proposed network are importantly different from other existing approaches.
Additional details for decision
The results from this paper are significantly inferior to the best results published. With a few quick searches, I found that there are several approaches performing better than the proposed model on every dataset considered in the analysis, as you can see below.
http://nlpprogress.com/english/text_classification.html
https://github.com/sebastianruder/NLP-progress/blob/master/english/sentiment_analysis.md
https://paperswithcode.com/sota/text-classification-on-yahoo-answers
Extra notes (not factoring in decision)
- Consider spacing out the 3 rightmost blocks in Figure 1, I found the layout confusing and there's space available.
- In section 3, I would have liked more explanation for the motivation of the various design choices.


Despite the impressive performance of deep neural networks (DNNs) on numerous learning tasks, they still exhibit uncouth behaviours. One puzzling behaviour is the subtle sensitive reaction of DNNs to various noise attacks. Such a nuisance has strengthened the line of research around developing and training noise-robust networks. In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input. We provide an efficient and simple approach to approximate such a regularizer for arbitrarily deep networks. This is done by leveraging the analytic expression of the output mean of a shallow neural network, avoiding the need for memory and computation expensive data augmentation. We conduct extensive experiments on LeNet and AlexNet on various datasets including MNIST, CIFAR10, and CIFAR100 to demonstrate the effectiveness of our proposed regularizer. In particular, we show that networks that are trained with the proposed regularizer benefit from a boost in robustness against Gaussian noise to an equivalent amount of performing 3-21 folds of noisy data augmentation. Moreover, we empirically show on several architectures and datasets that improving robustness against Gaussian noise, by using the new regularizer, can improve the overall robustness against 6 other types of attacks by two orders of magnitude.

This paper proposes a regularization method for achieving robustness to noisy inputs, with relatively less computation compared to standard data augmentation approaches. Specifically, the authors analyze the analytic expression of the loss on the noisy inputs, and using Jensen’s inequality, propose to minimize a surrogate loss over the expectation of noisy inputs. To minimize the loss over the expectation, the authors impose a regularization over the first moment of the network weights. The authors validate the model with the proposed regularization technique for its robustness against Gaussian attack and other types of attacks, whose results show that the model is robust.
Pros
- The general idea of the regularization that replaces the generation of noisy samples and optimization over it is conceptually appealing and seems practically useful.
- The derivation of the moment-based regularization makes sense.
- The proposed regularizer seems to be effective to a certain degree, on the sets of experiments done by the authors.
Cons
- Experimental validation seems highly inadequate due to lack of baselines. Thus it is difficult to assess the degree of robustness the proposed model achieves. The authors should perform extensive evaluation against state-of-the-art techniques against multiple types of attacks, in order to demonstrate the effectiveness of the proposed method.
- While the authors emphasize the computational efficiency of the method, the authors do not report computational cost or actual runtime.
- The types of non-Gaussian attacks should be better described. Which ones use L-infinity attacks and which use L2 attacks?
- Figure 3 doesn’t seem like a very favorable result to the proposed model, since we are generally more concerned with adversarial examples generated with small perturbations, as large perturbations may change the input semantics.
In sum, while I like the overall idea and find the work novel and potentially practical, it is difficult to properly evaluate the work due to lack of comparison against state-of-the-art data augmentation methods for achieving robustness. Therefore I temporarily give this paper a weak reject, but may change the rating with more experimental results provided in the rebuttal.


In many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target dataset -e.g., recognizing characters of a new font using a set of different fonts. While most recent research has considered ad-hoc combination rules to address this problem, we extend previous work on domain discrepancy minimization to develop a finite-sample generalization bound, and accordingly propose a theoretically justified optimization procedure. The algorithm we develop, Domain AggRegation Network (DARN), is able to effectively adjust the weight of each source domain during training to ensure relevant domains are given more importance for adaptation. We evaluate the proposed method on real-world sentiment analysis, digit recognition and object recognition datasets and show that DARN can significantly outperform the state-of-the-art alternatives.

###Summary###
This paper tackles the multi-source domain adaptation by aggregate multiple source domains dynamically during the training phase. The observation is that in many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target datasets.
Firstly, the paper derives a multiple-source domain adaptation upper-bound from single-to-single domain adaptation generalization bound, based on the theoretical work from Cortes et al (2019). The idea is similar to Zhao et al (2019), which introduces a weighted parameter \alpha to combine the source domains together.
Secondly, based on the theoretical result, the paper proposes an algorithm to minimize the upper bound of the theoretical result. The upper bound can be simplified as the quartic form (Eq. 4) and can be optimized with the Lagrangian form. Since no closed-form expression for the optimal v can be derived, the authors propose to use binary search to find it.
Based on the theoretical results and the algorithm, the paper introduces Domain AggRegation Network (DARN), which contains a base network for feature extraction, h_y to minimize the task loss and h_d to evaluate the discrepancy between each source domain and target domain.  The loss is aggregation with the parameter \alpha.
Finally, the paper conduct experiments on sentimental analysis benchmark, Amazon Review and digit datasets. The paper selects MDAN, DANN, MDMN as the baselines. On the amazon review dataset, the performance of the proposed DARN model is comparable with the MDMN baseline. On the digit dataset, the model can outperform the baselines.
### Novelty ###
The theoretical results in this paper are extended from Cortes et al (2019) and Zhao et al (2018).  Thus, the theoretical contribution of this paper is limited.
The algorithm proposed in this paper is interesting. However, the motivation of the proposed method is to minimize the upper bound, not the loss itself, i.e. L_T(h, f_T). Intuitively, when the upper bound of the loss is minimized, it will be beneficial to minimize the loss itself. But it's not guaranteed as the upper bound contains other variables, such as the number of training samples and model complexity. If the training samples and model complexity (think about the parameters in the deep models) are significantly large, the upper bound of the loss might be also very large.
As for the experimental results, the paper only provides results on the sentimental analysis results and digit datasets, which are small benchmarks. The selected baselines are not sufficient. The improvement from the baselines is also limited.
###Clarity###
Overall, the paper is well organized and logically clear. The images are well-presented and well-explained by the captions and the text.
The derivation of the algorithm in Sec 3.2 is logically clear and easy to follow.
###Pros###
1) The paper proposes a new theoretical upper-bound based on the prior works, the upper-bound and its derivation are interesting and heuristic to the domain adaptation research community.
2) The paper is applicable to many practical scenarios since the data from the real-world application is typically collected from multiple sources.
3) The paper is overall well-organized and well-written. The claims of the paper are verified by the experimental results.
###Cons###
1) The critical issue of this paper is that the algorithm is designed to minimize the upper bound. The idea is intuitive when the upper bound is small. However, the proposed upper bound in the paper involves other parameters, such as the model complexity and the number of training samples.
It's an intuitive idea to weight different source domains in multi-source domain adaptation. The paper derives the weight by the Lagrangian form to minimize the upper bound. While another trivial trick is to evaluate \alpha by the domain closeness between each source domain with the target domain.
2) The experimental results provided in this paper are weak. In the abstract and introduction,  the paper motivates the multi-source domain adaptation (MSDA) problem by arguing that the MSDA has a lot of real applications.  But the paper only provides empirical results on sentimental analysis and digit recognition.  Besides, the results on the sentimental analysis are comparable with the compared baselines.
It will be also interesting to see how does the proposed method perform on large-scale datasets such as DomainNet and Office-Home dataset:
DomainNet: Moment Matching for Multi-Source Domain Adaptation, ICCV 2019. http://ai.bu.edu/DomainNet/
Office-Home: Deep Hashing Network for Unsupervised Domain Adaptation, CVPR 2017. http://hemanthdv.org/OfficeHome-Dataset/
3) The novelty of this paper is incremental as the theoretical results are extended from Cortes et al (2019) and Zhao et al (2018).
Based on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.
To improve the rating, the author should explain the following questions:
1). Why minimizing the upper bound in this scenario would help to minimize the loss on the target domain, when the upper bound can be substantially huge? How about just evaluate \alpha by the closeness of the source domain with the target domain?
2). In the introduction, the paper motivates the multi-source domain adaptation (MSDA) problem by arguing that the MSDA has a lot of real applications. While the experiments are only performed on sentimental analysis and digit recognition. How about evaluating the proposed methods on real image recognition such as DomainNet or Office-Home?


Adversarial training, in which a network is trained on both adversarial and clean examples, is one of the most trusted defense methods against adversarial attacks. However, there are three major practical difficulties in implementing and deploying this method -expensive in terms of extra memory and computation costs; accuracy trade-off between clean and adversarial examples; and lack of diversity of adversarial perturbations. Classical adversarial training uses fixed, precomputed perturbations in adversarial examples (input space). In contrast, we introduce dynamic adversarial perturbations into the parameter space of the network, by adding perturbation biases to the fully connected layers of deep convolutional neural network. During training, using only clean images, the perturbation biases are updated in the Fast Gradient Sign Direction to automatically create and store adversarial perturbations by recycling the gradient information computed. The network learns and adjusts itself automatically to these learned adversarial perturbations. Thus, we can achieve adversarial training with negligible cost compared to requiring a training set of adversarial example images. In addition, if combined with classical adversarial training, our perturbation biases can alleviate accuracy trade-off difficulties, and diversify adversarial perturbations.

This paper proposes perturbation biases as a counter-measure against adversarial perturbations. The perturbation biases are additional bias terms that are trained by a variant of gradient ascent. The method imposes less computational costs compared to most adversarial training algorithms. In their experimental evaluation, the algorithm achieved higher accuracy on both clean and adversarial examples.
This paper should be rejected because the proposed method is not well justified either by theory or practice. Experiments are weak and do not support the effectiveness of the proposed method.
Major comments:
Since the evaluations of defense algorithms are often misleading [1], it requires throughout experiments or theoretical certifications to confirm the effectiveness of defense methods. However, the experiment configuration in this paper is not satisfactory to demonstrate the robustness of defended networks. The followings are a list of concerns.
1) Experiments are limited to small datasets and networks. Since some phenomena only appear in larger datasets [2], there is a concern that the proposed method also works on other datasets.
2) The attack algorithm used for the evaluation is weak. We can confirm this by observing the label leakage [2] in the experimental results. It is hard to judge which defenses are most effective, even within the tested datasets and models.
3) The "adversarial training" baseline used in the experiment is weird. Adversarial training typically generates adversarial examples during the process of the neural networks' optimization instead of using precomputed adversarial examples. Baseline methods should be stronger, for example, adversarial training with PGD [3].
[1] Athalye et al. "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples." ICML 2018
[2] Kurakin et al. "ADVERSARIAL MACHINE LEARNING AT SCALE." ICLR 2017
[3] Madry et al. "Towards Deep Learning Models Resistant to Adversarial Attacks." ICLR 2018


Predicting the emotional impact of videos using machine learning is a challenging task. Feature extraction, multi-modal fusion and temporal context fusion are crucial stages for predicting valence and arousal values in the emotional impact, but have not been successfully exploited. In this paper, we proposed a comprehensive framework with innovative designs of model structure and multi-modal fusion strategy. We select the most suitable modalities for valence and arousal tasks respectively and each modal feature is extracted using the modality-specific pretrained deep model on large generic dataset. Two-time-scale structures, one for the intra-clip and the other for the inter-clip, are proposed to capture the temporal dependency of video content and emotional states. To combine the complementary information from multiple modalities, an effective and efficient residual-based progressive training strategy is proposed. Each modality is step-wisely combined into the multi-modal model, responsible for completing the missing parts of features. With all those above, our proposed prediction framework achieves better performance with a large margin compared to the state-of-the-art.

This paper tackles the problem of affective impact prediction from multimodal sequences. The authors achieve state-of-the-art performance by using (i) a two-time-scale temporal feature extractor, (ii) progressive training strategy for multi-modal feature fusion, and (iii) pretraining. They divide a long video sequence evenly into several clips. The idea of applying one LSTM for intra-clips (short-time) temporal feature extraction and another LSTM for inter-clips (long-time) temporal feature extraction, resulting in two-time-scale, looks reasonable choice but not novel. The proposed progressive training strategy is mainly used for the modality-specific LSTMs training, which is used during the intra-clip short-time modeling phrase. It seems working well in practice. However, from the explanation, it's not clear why this strategy is good for the *complementary* fusion. Each modality LSTM is trained sequentially with features extracted from other fixed modality LSTMs. Also, the authors seem not explaining why they set the training order for LSTMs at this stage to be the descending order of their performance in the previous stage. In the experiments section, two weak baseline models and several previous state-of-the-art models are compared. However, enough ablation studies on the two-time-scale structure and proposed training strategy are not provided to demonstrate that the proposed method does provide complementary multi-modal feature fusion, which is claimed as a contribution.
Overall, although it achieves state-of-the-art performance on the task, none of the claimed contributions is novel or significant enough. It is a combination of existing ideas (but giving good performance). The proposed training procedure is also weak to be considered as a scientific contribution.


The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible code can be found on http://anonymized

In this work, the authors show that the sequence of self-attention and feed-forward layers within a Transformer can be interpreted as an approximate numerical solution to a set of coupled ODEs. Based on this insight, the authors propose to replace the first-order Lie-Trotter splitting scheme by the more accurate, second-order Strang splitting scheme. They then present experimental results that indicate an improved performance of their Macaron Net compared to the Transformer and argue that this is due to the former being a more accurate numerical solution to the underlying set of ODEs.
The authors highlight an interesting connection between the Transformer architecture and ODEs. In particular, they derive a set of ODEs that is solved numerically by the Transformer and borrow from the body of literature on numerical ODE solvers to improve the architecture. I find that this is a very elegant and promising approach for finding better architectures.
However, I also identified two major and a couple of minor shortcomings of the paper that are explained in detail below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.
Major points:
1) Replacing the first-order operator splitting scheme by a second-order scheme only guarantees a lower overall truncation error if the split ODEs are solved with sufficiently high accuracy. In particular, the overall accuracy of the numerical solution to the original ODE depends on the accuracy of the operator splitting and the accuracy of the integration scheme used to solve the split ODEs (e.g. Euler’s method). The authors improve the operator splitting, i.e. they make it second-order, but they keep Euler’s method to integrate the individual ODEs. Because of that, they actually do not get rid of the lowest-order error term of the overall scheme and therefore do not obtain a more accurate ODE solver. I think this is a crucial point that invalidates the authors’ claim that the Macaron Net employs a higher-order integration scheme. As far as I am aware, this is not commented on in the paper at all. To address this shortcoming, the authors could replace Euler’s method by a second-order integrator.
2) The experiments considered in this paper are interesting and show competitive performance but, in my opinion, they do not sufficiently support the claim that the Macaron Net yields a more accurate solution to the underlying set of ODEs compared to a Transformer. For a more convincing support of this claim, the authors could consider a toy problem, i.e. a simple set of ODEs with known analytical solution, and actually show that the Macaron Net is more accurate. The accuracy of a numerical ODE solver is commonly assessed by plotting the absolute difference between the exact solution (or a high-resolution numerical approximation to it) and the numerical solution vs the timestep (here \gamma). I suspect that such an analysis would support my previous comment and show that the proposed new architecture is not more accurate ODE solver than the original one.
Minor points and questions:
i) Eqs. (17-19) suggest that you apply two different FFN layers (doubling the number of parameters) instead of applying the same FFN layer twice. You comment on this in Sec. 4.1 when you say ‘..., we set the dimensionality of the inner-layer of the two FFN sub-layers in the Macaron layers to two times of the dimensionality of the hidden states’. It is not clear to me why consistency with Strang splitting requires two different layers rather than applying the same FFN layer twice. Is the reason for having a separate, trainable layer to account for the explicit time dependence of G in Eq. (16)? I think that this is a very important point that should be clarified.
ii) I think that this type of system is usually referred to as ‘dynamical system’ and not ‘dynamic system’. Please check that and, if applicable, update the title.
iii) The authors say that Eq. (5) is a 'convection-diffusion equation’. As far as I am aware, the diffusion equation is a partial differential equation (PDE). Perhaps there is a different notion 'diffusion equation’ in ODE theory. If that’s the case, could the authors please clarify this point to avoid confusion, e.g. by adding a suitable reference in which this type of ODE is classified as a convection-diffusion equation?
iv) In Sec. 2 (2nd paragraph),  you cite Vaswani et al. (2017) but in that work the quantity under the square-root in the denominator of Attention(Q, K, V) is actually d_k, the dimension of the key, and not d_model.
v) Figure 1 is a very vague illustration of the connection to ODEs and provides almost no explanation in the caption. I don’t think there is much value in having this figure there.
vi) There are a couple of mistakes in the paper (grammar and expressions) that should be fixed. For example, ‘the Euler’s method’ instead of ‘Euler’s method’, ‘movement in the space’ instead of ‘movement in space’, ‘dynamic system’ instead of ‘dynamical system’, ‘project parameter matrices’ instead of ‘parameter matrices’ or ‘projections’, ‘specially’ instead of ‘specifically’, etc. Please take a look at the relevant sections in the paper and revise them accordingly.
vii) You explain multiple times why the proposed architecture is called a ‘Macaron Net’ (Abstract, Sec 1, Sec. 3). To avoid repetition, I would only explain it once.


We investigate global adversarial robustness guarantees for machine learning models. Specifically, given a trained model we consider the problem of computing the probability that its prediction at any point sampled from the (unknown) input distribution is susceptible to adversarial attacks. Assuming continuity of the model, we prove measurability for a selection of local robustness properties used in the literature. We then show how concentration inequalities can be employed to compute global robustness with estimation error upper-bounded by , for any > 0 selected a priori. We utilise the methods to provide statistically sound analysis of the robustness/accuracy trade-off for a variety of neural networks architectures and training methods on MNIST, Fashion-MNIST and CIFAR. We empirically observe that robustness and accuracy tend to be negatively correlated for networks trained via stochastic gradient descent and with iterative pruning techniques, while a positive trend is observed between them in Bayesian settings.

This paper seeks to analyze the global robustness of neural networks, a concept defined in the paper. The authors show using concentration inequalities that the empirical local robustness approximates the global robustness. The authors investigate various other issues in the robustness literature, including the robustness/accuracy tradeoff, whether iterative pruning increases robustness, and the robustness of Bayesian networks.
I would vote for rejecting this paper for two key reasons. First, the notion of global robustness is not well-motivated (why do we want to compute this metric? What does it tell us that local robustness does not?). Second, the paper tries to do too many different things, and as a result does not give enough attention to any particular topic.
First, I believe it is up to the authors to motivate their study of global robustness further. While I acknowledge that a few prior works exists along these lines, I do not feel that this work provides much new insight into why global robustness is interesting to examine.
The authors go on to prove results showing that an empirical estimator of the local robustness will converge to the global robustness. The bounds require that the dataset size scales with eps^-2, where eps is the error. This is not terrible but also not great; for example, achieving 1% error requires a dataset size of 10^4 (realistically, even larger datasets would be required to achieve results with high probability).
Next, I would suggest that the authors avoid using the word “guarantees” if they are estimating empirical local robustness in an approximate (rather than exact) manner. Guarantees implies strict results, but the authors use a weak attack (FGSM) to approximate empirical local robustness. The results from FGSM could be far from optimal; the authors could use a stronger attack (e.g. PGD) in addition to changing the wording, or they could find provable guarantees using alternate methods.
Lastly, the authors try to tackle 3 extra questions beyond global robustness toward the end of the paper, and the last two questions are not properly fleshed out.
I like section 4.2, where the authors empirically show that networks that have better hyperparameters (for regular accuracy) tend to be less robust. This is a confirmation of a previously studied phenomena in the literature. Ideally, I would also appreciate it if the authors found the line of best fit to the dataset in addition to the plots provided. I would like a clarification on whether any of these networks were trained to be robust, although it appears that they were all trained normally. I would also like to see plot 2c (for the standard case of robustness of C(x_tilde) = C(x)), except for MNIST and CIFAR10 as well. I feel that the last-layer representation metric the authors analyze (f(x) is close to f(x_tilde)) could be misleading, as robustness on the last layer does not necessarily imply standard adversarial robustness.
Section 4.3 explores iterative pruning, but that seems fairly unrelated to the rest of the paper. Finally, Section 4.4 tries to show the opposite trend for Bayesian Neural Networks, but unfortunately the results for such networks do not yet scale beyond MNIST.
Additional Feedback:
- Why did you use R^emp and D^emp as opposed to just R^emp(g) and R^emp(g_bar)?
- In Figure 1, what is the dataset size |S|?
- In the last sentence of Section 4.3, I didn’t understand what you meant about the relationship between weight pruning and network regularization. Do you mean that weight regularization has no effect on robustness, just like iterative weight pruning?


The performance of deep neural networks is often attributed to their automated, taskrelated feature construction. It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples. Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error. For several flatness measures, this correlation has been empirically validated. However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization: if a network uses ReLU activations, the network function can be reparameterized without changing its output in such a way that flatness is changed almost arbitrarily. This paper proposes a natural modification of existing flatness measures that results in invariance to reparameterization. The proposed measures imply a robustness of the network to changes in the input and the hidden layers. Connecting this feature robustness to generalization leads to a generalized definition of the representativeness of data . With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.

This paper describes a connection between flatness of minima and generalization in deep neural networks. The authors define a concept called "feature-robustness" and show that it is related to flatness. This is derived through a straightforward observation that perturbations in feature space can be recast as perturbations of the model in parameter space. This allows the authors to define a (layerwise) flatness measure for minima in deep networks (this layerwise flatness measure is also invariant to rescalings of the layers in neural networks with positively homogenous activations). The authors combine their notion of feature robustness with epsilon representativeness of a function to connect flatness to generalization. They present a few empirical evaluations on CIFAR10 and MNIST.
I believe this paper is able to once again confirm the relationship between flatness and generalization in an empirical manner with their layerwise measure of flatness. I am not so convinced about the theoretical justification that they claim to provide and thus do not recommend acceptance.
Theory - The key theorem relating generalization and flatness is Theorem 10 which says that if a compositional model is feature robust and the output of the first component is an epsilon-representative for the second component, then the compositional model will generalize. While this is interesting, it is not clear to me that this guarantees generalization for deep neural networks. This result only talks about feature robustness and representativeness for a particular layer. If a deep network has many layers, will the feature robustness layers closer to the input guarantee feature robustness at deeper layers? That might require a further unit operator norm constraint on the layer operator, which is a restriction on the types of weights that can be used. If a sample is epsilon representative at one layer, what is required for the next layer to be epsilon representative for the rest of the deep network? This seems to be a missing step in relating flatness/feature robustness of a layer to the generalization of the whole network.
Another idea that I think arises from Theorem 10 is that the flatness of loss landscapes is important when you have learning problems where the hypothesis class is compositional. While flatness is only spoken of in the case of deep neural networks, can we identify the same phenomenon in other problems? I would encourage the authors to try and identify another model in which the flatness-generalization relationship exists (even empirical evidence would suffice for now). This would strengthen the case for studying flatness and biasing optimization towards flatter solutions in the case of deep networks.
Experiments - This section seems to be pretty rudimentary, I would like to see more results on different kinds of network architectures (VGG? Inception? AlexNet?), more datasets (KMNIST? Fashion MNIST? SVHN?), and possibly more repetitions. At one point the authors mention that they declare a minimum has been reached if the training loss is < 0.07. Atleast on CIFAR10 and MNIST it is possible to achieve training loss <1e-4 so am not sure if the networks that the authors are testing are minima at all (It is important for them to be minima since the flatness measure is only defined at minima). Can the authors also identify more situations other than large batch vs small batch training that would lead them to obtain flatter/sharper minima?
The authors also claim that measuring generalization using test error is flawed, but do not provide details about their method of measuring generalization. I would want to see these details and a more thorough discussion of why measuring generalization through test error is flawed.
While this is an interesting paper, I do not believe it is ready for acceptance at ICLR 2020.


Achieving faster execution with shorter compilation time can foster further diversity and innovation in neural networks. However, the current paradigm of executing neural networks either relies on hand-optimized libraries, traditional compilation heuristics, or very recently genetic algorithms and other stochastic methods. These methods suffer from frequent costly hardware measurements rendering them not only too time consuming but also suboptimal. As such, we devise a solution that can learn to quickly adapt to a previously unseen design space for code optimization, both accelerating the search and improving the output performance. This solution dubbed CHAMELEON leverages reinforcement learning whose solution takes fewer steps to converge, and develops an adaptive sampling algorithm that not only focuses on the costly samples (real hardware measurements) on representative points but also uses a domain-knowledge inspired logic to improve the samples itself. Experimentation with real hardware shows that CHAMELEON provides 4.45×speed up in optimization time over AutoTVM, while also improving inference time of the modern deep networks by 5.6%.

This paper proposes an optimizing compiler  for DNN's based on adaptive sampling and reinforcement learning, to drive the search of optimal code in order to reduce compilation time as well as potentially improve the efficiency of the code produced. In particular, the paper proposes to use PPO to optimize a code optimization "search" policy, and then use K-mean clustering over a set of different proposed compilation proposals, from which to perform adaptive sampling to reduce compilation time while still keeping a high diversity of the proposed solution pools during exploration. At the same time the authors claim that using RL will learn a better search strategy compared to random search - such as simulated annealing which is used by competing methods - thus producing faster and better solutions.
The paper show results of up to 4x speedup in compilation time (autotuning time) while obtaining a slightly better or similar efficiency of the generated code (in term of execution time). This is a well written extensive research with good results. The authors mention it is (will be) integrated in the open source code of TVM. However I could find no mention in the paper of whether the code will be released with this publication, and I would like to solicit the authors to clarify their code release strategy and timing.
My other question pertains to whether or not  compilation time is an key metric to target. It is important to some extent, but I would say that aside from exponential / super-polynomial behaviour of auto-tuning algorithms, a multiple hours / days process to create the best optimized code for a certain network / hardware platform might not be such a big hurdle for a community already used to multiple days / weeks / months to train the same models. I believe that focusing on the efficiency of the optimized code produced would probably be a better metric of success.


As deep neural networks (DNNs) achieve tremendous success across many applica-1 tion domains, researchers tried to explore in many aspects on why they generalize 2 well. In this paper, we provide a novel perspective on these issues using the gradient 3 signal to noise ratio (GSNR) of parameters during training process of DNNs. The 4 GSNR of a parameter is defined as the ratio between its gradient's squared mean and 5 variance, over the data distribution. Based on several approximations, we establish 6 a quantitative relationship between model parameters' GSNR and the generaliza-7 tion gap. This relationship indicates that larger GSNR during training process leads 8 to better generalization performance. Moreover, we show that, different from that 9 of shallow models (e.g. logistic regression, support vector machines), the gradient 10 descent optimization dynamics of DNNs naturally produces large GSNR during 11 training, which is probably the key to DNNs' remarkable generalization ability.

In this work, the authors suggest a new point of view on generalization through the lens of the distribution of the per-sample gradients. The authors consider the variance and mean of the per-sample gradients for each parameter of the model and define for each parameter the Gradient Signal to Noise ratio (GSNR). The GSNR of a parameter is the ratio between the mean squared of the gradient per parameter per sample (computed over the samples) and the variance of the gradient per parameter per sample (also computed over the samples). The GSNR is promising as a measure of generalization and the authors provide a nice leading order derivation of the GSNR as a proxy for the measure of the generalization gap in the model. After the derivation, experimental results on MNIST are presented and suggest that empirically there is a relation between the generalization gap of neural network trained by gradient descent and the GSNR quantity given in the paper. Next the author analyze the GNSR of DNNs as opposed with shallow models or other learning techniques and observe that the GSNR differs when using random labels (lower GSNR) as compared with true labels and exhibits different behavior along training for DNNs and gradient descent.
 Assumption 2.3.1 is not necessarily the most realistic in the current training paradigm since multi-epoch training indeed separates the training and testing per-sample gradient distributions significantly.
Overall the paper gives a fresh (as far as I know) and nice idea on generalization of neural networks. The derivation requires quite a few stringent assumption (the leading order analysis, on the step size, assumption 2.3.1 on the test and train gradient distributions) the experiments do suggest that the theory is valid to an extent, especially during the early parts of training. In contrast, the presentation of the paper distracts from the work and needs additional cleaning up. Also, the experimental section 2.4 would benefit additional empirical analysis on other datasets  and additional experiments, as well as more thorough explanation of the experiments. The writing of the paper needs additional proofreading as currently it is easy to spot typos and grammar errors along the paper. I currently vote weak reject, for solid content and not-quite-ready presentation.
Additional experiments and careful proofreading should definitely enhance the paper and get it to the level of publications, so I am willing to change my decision if the authors improve the writing and the overall presentation. I like the idea presented in the paper and encourage the authors to resubmit a more tidy draft.
smaller presentation issues and typos/grammar issues:
Figure 1: name X and Y labels with more meaningful names instead of RHS,LHS
Figure 3(c): add legend
line 64: consists >>> which consists
line 73: we refer R(z) >>> we refer to R(z)
line 79 futher >>> further overall sentence needs more work
line 92 distribution becomes >>> distributions become
line 100: using common notation summing over n samples, the sum should possibly start from i=1 to n instead of i = 0 to n
line 132 include >>> includes (also possibly rephrase sentence)
line 136 vefity >>> verify
line 137: M number >>> M
line 157 the data points closely distributed >>> the data points are closely distributed
line 162 thorough analytically >>> thorough analytical

-------- Update After Rebuttal --------
I have read the comments the authors made and reviewed the paper's revision.
The presentation has improved a lot (Even though there is still room to go).
Nonetheless at this time I choose to update my score to a weak accept, as I think the authors bring a fresh (as far as I know) and interesting idea with regards to empirical generalization.


Stereo matching is one of the important basic tasks in the computer vision field. In recent years, stereo matching algorithms based on deep learning have achieved excellent performance and become the mainstream research direction. Existing algorithms generally use deep convolutional neural networks (DCNNs) to extract more abstract semantic information, but we believe that the detailed information of the spatial structure is more important for stereo matching tasks. Based on this point of view, this paper proposes a shallow feature extraction network with a large receptive field. The network consists of three parts: a primary feature extraction module, an atrous spatial pyramid pooling (ASPP) module and a feature fusion module. The primary feature extraction network contains only three convolution layers. This network utilizes the basic feature extraction ability of the shallow network to extract and retain the detailed information of the spatial structure. In this paper, the dilated convolution and atrous spatial pyramid pooling (ASPP) module are introduced to increase the size of receptive field. In addition, a feature fusion module is designed, which integrates the feature maps with multiscale receptive fields and mutually complements the feature information of different scales. We replaced the feature extraction part of the existing stereo matching algorithms with our shallow feature extraction network, and achieved state-of-the-art performance on the KITTI 2015 dataset. Compared with the reference network, the number of parameters is reduced by 42%, and the matching accuracy is improved by 1.9%.

The paper at hand argues that shallow feature extraction networks should be favored for the computer vision task of stereo matching, rather than the commonly used deep ResNet backbones. To that end, a model is proposed consisting of three convolutional feature extraction layers only. As this makes it impossible to capture global context, dilated convolutions with varying dilations are applied to the extracted features in parallel and concatenated. Finally, a fusion module implemented via channel attention is applied.
I found this paper lacking in terms of contributions. While the motivation for retaining detailed feature information makes sense for the task in question, it is not clear why it requires replacing the ResNet feature extraction with a very shallow network; an alternative would be to add skip-connections originating from lower levels. Both dilated convolutions and the fusion module are not novel. The paper addresses the need for a fusion module in great detail. However, the insight that a large dilation on top of features computed from a limited receptive field will result in a non-continuous receptive field is a rather trivial one. Hence I don't see the need for section 4.3.1 and 4.3.2, as well as the somewhat complicated deduction in appendix A. The individual modules are ablated, but what about the number of layers for feature extraction? Why settle for 3 rather than 1, 2, or 4?
The paper does compare against other methods for stereo matching, but I was wondering why the KTTI leader-board excerpt does not include the better results from http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo ? The result tables are also hard to parse as bold numbers do not correspond to best performance. The conclusion claims that the proposed shallow architecture exhibits "lower training difficulty" but I did not see any support for this in the experiments.
Overall, I think that this paper should be rejected on the basis of insufficient contributions.


We explore the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data. Challenges arise in practice. As existing state-of-the-art algorithms are tailored to certain statistical models, we have different best algorithms across distinct scenarios. Worse yet, we have no prior knowledge on the underlying model for a given scenario. These call for a unified approach that can be universally applied to a wide range of scenarios and achieve consistently high performances. To this end, we incorporate deep learning architectures so as to reflect the key structural features that most state-of-the-art algorithms, some of which are optimal in certain settings, share in common. This enables us to infer hidden models underlying a given dataset, which govern in-group interactions and statistical patterns of comparisons, and hence to devise the best algorithm tailored to the dataset at hand. Through extensive experiments on synthetic and real-world datasets, we evaluate our framework in comparison to state-of-the-art algorithms. It turns out that our framework consistently leads to the best performance across all datasets in terms of cross entropy loss and prediction accuracy, while the state-of-the-art algorithms suffer from inconsistent performances across different datasets. Furthermore, we show that it can be easily extended to attain satisfactory performances in rank aggregation tasks, suggesting that it can be adaptable for other tasks as well.

This paper provides a technique to solve match prediction problem -- the problem of estimating likelihood of preference between a pair of M-sized sets. The paper replaces the previously proposed conventional statistical models with a deep learning architecture and achieve superior performance than some of the baselines. The experiments show the efficacy of the proposed methods.
I have some major concerns with this paper. These are described below:
1.  The paper presentation is not very clear. The paper contains many imprecise statements. The problem is not setup very well.
(a) The abstract and the introduction  contains vague statements with no proper description of what the technique is about.
(b) In introduction, the authors start with discussing 1-sized pairwise comparisons, and then suddenly are discussing in-group items effects at the end of the third paragraph, so that means they are talking about M-sized comparisons with M > 1. This adds to the confusion.
(c) The same things holds true for the first paragraph in "Main Contribution". Since the authors are using deep learning frameworks, this does not mean the authors "can infer the underlying models accurately." I am not sure what the authors wanted to convey with this statement.
(d) The reason I am being very stringent with impreciseness of the statements is majorly after reading the first statement in the Motivation section. It says, "Our decision to incorporate two modules R and P into our architecture has been inspired by SOME state-of-the-art algorithms developed under CERTAIN statistical models, which have been shown
therein to be OPTIMAL." Please avoid using some, certain, optimal when they are not defined properly in the paper yet.

 2. The paper mentions multiple times that it does not use statistical models tailored for a dataset or application but instead use deep neural networks. In my opinion, NN is just another form of statistical model that captures statistical patterns of comparisons and in-group interaction.  They are not following the same modeling assumptions as others but have created their own in some sense. The authors may want to rephrase those statements.
3. The evaluation metric for the datasets that author consider is the prediction accuracy. I am not sure why the authors evaluate on cross entropy as well in Table 1, since both are closely related (CE is a consistent surrogate of accuracy). Can you please explain? However, I liked that they compared the methods with other metrics in Table 2.
4. In problem setup, please mention M > 1. For M = 1, the problem is similar to [1] and many solutions have been proposed for that.
5. I am not sure how the R and P modules capture what the authors want them to capture. I would suggest the authors to include that when they first discuss R and P modules.
6. In equations 7, it is not clear how the function R(.,.,.,) and P(.,.,.) defined? Is it similar to what is described in equation 5?
7. The training procedure contains the standard details; however, it is not clear to me how the modules R, P, and G interact during training.
Overall, I believe the paper has a descent idea and contains satisfactory experimental results; however, the presentation of the paper is very weak at this moment.
[1] Joachims, Thorsten. "Optimizing search engines using clickthrough data." Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2002.
---- After Rebuttal ---
I thank the authors for providing response to my questions and making edits to the paper. My clarity on boxes R and G has become better; however, I am still not totally convinced. Also, the presentation of the paper could be further improved. Therefore, I would keep the same score.


As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.

The authors present a framework to perform meta-learning on the loss used for
training. They introduce the Baikal loss, obtained using the MNIST dataset, and
BaikalCMA where the coefficients have been tuned. The evaluation of these loss
functions is performed on the MNIST and CIFAR-10, and according to the results
they converge faster, towards lower test error and need fewer samples to obtain
results similar to the cross-entropy loss.
The claims are clearly stated and the framework is detailed, the experiments
cover all the potential benefits of the Baikal loss. However it seems that some
potentially critical points have been omitted. The cross-entropy loss is well
known to be beneficial in dataset with severe class imbalance. The two datasets
used for evaluation are perfectly balanced, it might beneficial to see how it
performs in the unbalanced case.
I have a couple of concerns about the method. First about step "(1) loss
function discovery": The initial population starts with trees of depth at most
2, and the final solution(Baikal) has either 2 or 3 (depending on which
definition of depth is chosen). It is unclear that the genetic optimization is
superior to simply choosing random loss functions. I think it would be relevant
to add a figure that shows how the fitness of the leader of each generation
evolves over time.
The second step "(2) coefficient optimization", while objectively generating a
loss function that was superior on the metrics evaluated, raised some
questions. In equation (2) the factor "1.5352" seems to be equivalent to adding
a constant to the loss, which should not impact optimization. Also the factor
"2.7279" seems to be equivalent to a change in learning rate. This may be an
indication that the learning rate search was not done thoroughly. It would be
beneficial to clarify when it is happening: a) For each individual of the
population during step (1), b) before performing CMA, c) after CMA. Also: Was
learning rate search was performed on the network trained with Cross-Entropy? It
was not entirely clear from the experiment details in Appendix A.2.1.
About the Baikal loss itself, I fear that it could produce models that have very
poor calibration, it might be nice to evaluate that (even if it is only in
the appendix).
While the paper does a great job at presenting the problem and its applications
and propose a framework that generated a loss that can transfer to other
datasets without any tuning required. I think it lacks a more thorough
evaluation and description of the dynamics observed during the genetic
evolution, and the performance of the Baikal loss on other datasets (my quick
experients with it on ImageNet diverged I did not have the time necessary to
tune the hyper-parameters).
Minor remarks:
There might be a slight omission in section 3.1: according to Figure 1, exp(x)
is one of the potential unary operators explored by the GLO framework. However
it is not present it the list of operators. Could you clarify this?
To the best of my knowledge, in the machine learning literature, it seems that
the letter x is used to denote the prediction and y for the ground truth. The
fact that this paper used the opposite convention confused me the first time I
read it.