Most state-of-the-art neural machine translation systems, despite being different in architectural skeletons (e.g., recurrence, convolutional), share an indispensable feature: the Attention. However, most existing attention methods are token-based and ignore the importance of phrasal alignments, the key ingredient for the success of phrase-based statistical machine translation. In this paper, we propose novel phrase-based attention methods to model n-grams of tokens as attention entities. We incorporate our phrase-based attentions into the recently proposed Transformer network, and demonstrate that our approach yields improvements of up to 1.3 and 0.5 BLEU points in English-to-German and German-to-English translation tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-toEnglish translation tasks on WMT'14 test set using WMT'16 training data.

The authors propose to include phrases (contiguous n-grams of wordpieces) in both the self-attention and encoder-decoder attention modules of the Transformer model (Vaswani et al., 2017). In standard multi-head attention, the logits of the attention distribution of each head is computed as the dot-product between query and key representations, which are position-specific. In the phrase-based attention proposed here, a convolution is first computed over the query, key, value sequences before logits are computed as before (a few variants of this scheme are explored). Results show an improvement of up to 1.3 BLEU points compared to the baseline Transformer model. However, the lack of a controlled experiment sheds substantial doubt on the efficiacy of the model (see below).
Contributions
-------------------
Proposes a simple way to incorporate n-grams in the Transformer model. The implementation is straightforward and should be fully replicable in an afternoon.
Having an inductive bias towards modeling of longer phrases seems intuitively useful, in particular when using subword representations, where subword units are often ambiguous. This is also motivated by the fact that prior work has shown that subword regularization, where sampling different subword segmentations during training can be useful.
Improvements in BLEU scores are quite strong.
Issues
---------
The experiments do not control for parameter count. The phrasal attention model adds significant number of parameters (e.g., "interleaved attention" corresponds to 3x the number of 1D convolution parameters in the attention layer). It is well established that more parameters correspond to increased BLEU scores (e.g., the 2x parameter count in the "big" Transformer setting from Vaswani et al. (2017) results in over 1 BLEU point improvement). This needs to be fixed!
The model is a very modest extension of the original Transformer model and so its value to the community beyond improved numbers is somewhat questionable.
While an explicit inductive bias for phrases seem plausible, it may be that this can already be fully captured by multi-head attention. With positional encodings, two heads can easily attend to adjacent positions which gives the model the same capacity as the convolutional phrase model. The result in the paper that trigrams do not add anything on top of bigrams signals to me that the model is already implicitly capturing phrase-level aspects in the multi-head attention. I would urge the authors to verify this by looking at gradient information (https://arxiv.org/abs/1312.6034).
There are several unsubstantiated claims: "Without specific attention to phrases, a particular attention function has to depend entirely on the token-level softmax scores of a phrase for phrasal alignment, which is not robust and reliable, thus making it more difficult to learn the mappings." - The attention is positional, but not necessarily token-based. The model has capacity to represent phrases in subsequent layers. WIth h heads , a position in the k-th layer can in principle represent h^k grams (each slot in layer 2 can represent a h-gram and so on).
The differences in training setup compared to Vaswani et al. (2017) needs to be explicit ("most of the training settings" is too handwavy). Please list any differences.
The notation is somewhat cumbersome and could use some polishing. For example, the input and output symbols both range over indices in [1,n]. The multi-head attention formulas also do not match the ones from Vaswani et al. (2017) fully. Please ensure consistency and readability of symbols and formulas.
The model inspection would be much improved by variance analysis. For example, the numbers in table 3 would be more useful if accompanied by variance across training runs. The particular allocation could well be an effect of random initialization. I could also see other reasons for this particular allocation than phrases being more useful in intermediate layers (e.g., positional encodings in the first layer is a strong bias towards token-to-token attention, it could be that the magnitude of convolved vectors is larger than the batch-normalized unigram encodings, so that logits are larger.
Questions
--------------
In "query-as-kernel convolution", it is unclear whether you map Q[t, :] into n x d_q x d_k convolution kernel parameters, or if each element of the window around Q[t] of width n is mapped to a convolution kernel parameter. Also what is the exact form of the transformation. Do you transform the d_q dimensional vectors in Q to a d_q x d_k matrix? Is this done by mapping to a d_q * d_k dimensional vector which is then rearranged into the convolution kernel matrix?
Does the model tend to choose one particular n-gram type for a particular position, or will it select different n-gram types for the same position?
"The selection of which n-gram to assign to how many heads is arbitrary" - How is this arbitrary? This seems a rather strong inductive bias?
"However, the homogeneity restriction may limit the model to learn interactions between different n-gram types" - How is the case? It seems rather that the limitation is that the model cannot dynamically allocate heads to the most relevant n-gram type?
I do not understand equation 14. Do you mean I_dec = I_cross = (...)?
"Phrase-to-phrase mapping helps model local agreement, e.g., between an adjective and a noun (in terms of gender, number and case) or between subject and verb (in terms of person and number)." Is this actually verified with experiments / model inspection?
"This is especially necessary when the target language is morphologically rich, like German, whose words are usually compounded with sub-words expressing different meanings and grammatical structures" This claim should be verified, e.g. by comparing to English-French as well as model inspection.


This paper provides theoretical analysis of generative adversarial networks (GANs) to explain its advantages over other standard methods of learning probability measures. GANs learn a probability through observations, using the objective function with a generator and a discriminator. While many empirical results indicate that GANs can generate realistic samples, the reason for such successful performance remains unelucidated. This paper focuses the situation where the target probability measure satisfies the disconnected support property, which means a separate support of a probability, and relates it with the advantage of GANs. It is theoretically shown that, unlike other popular models, GANs do not suffer from the decrease of generalization performance caused by the disconnected support property. We rigorously quantify the generalization performance of GANs of a given architecture, and compare it with the performance of the other models. Based on the theory, we also provide a guideline for selecting deep network architecture for GANs. We demonstrate some numerical examples which support our results.

This paper provides a theoretical study of GANs in the following setting:
- The target distribution has a locally smooth density on a compact set [0, 1]^D. It might be supported only on M disjoint components, each of which has a smooth boundary, within that compact set.
- The latent noise dimension (inputs to the generator) is of the same dimension as the data.
- An IPM loss (1) is used.
- The discriminator functions of the IPM and the generator networks are both ReLU networks of at most L layers and at most S total nonzero weights, with all weights having magnitude at most B.
- The discriminator functions are Lipschitz continuous; this is implied by the previous assumption, but the bound is tighter if we have a tighter constraint here.
- We obtain the generator which exactly minimizes the IPM between the empirical distributions of m samples from the model and n samples from the target.
- Maybe: there is some g such that P* is produced by g. It's not clear in your statement of Theorem 1 whether this is necessary, or exactly what it means when you say that in Lemma 3, and I haven't fully checked what's used yet; it would help to explicitly say this is not assumed if it's not.
Pointing out that the kinds of distributions handled by GANs often have disjoint support, and analyzing this case, is certainly of interest.
The full-dimensional support assumption is not ideal, but of course just because the analysis doesn't apply to most practical GAN settings doesn't mean it's not an important step towards one that does.
Also note that the assumption about the structure of the networks eliminates the MMD GANs that you use in experiments -- which have a kernel function at the top of the critic network -- though it does allow for most GAN variants. Maybe the most interesting algorithm for this setting is the Coulomb GAN (ICLR 2018, https://arxiv.org/abs/1708.08819 ), which uses a neural network critic of the kind you study but estimates a distance which (unlike the Wasserstein and most other GAN objective functions) has good statistical convergence properties (kappa=2 as you mention).
My biggest concern by far, though, is Proposition 1. You present it as if it's a lower bound: establishing that there is some class of distributions for which Theorem 1 shows that GANs can account for local smoothness and standard methods are shown not to be able to. This isn't what you do; instead you exhibit a class of distributions for which Theorem 1 shows that GANs can account for local smoothness, and previous analyses of standard methods do not show that they are able to take advantage of it. This is not the same thing at all! Although the previous upper bounds have matching lower bounds, you don't demonstrate (and it is likely not the case) that the distribution you show fits into the class of distributions used by the previous lower bounds, and so it remains very possible that other methods are able to take advantage of local smoothness as well as GANs do.
Given that Proposition 1 is hence a very weak statement, your main contribution in terms of disconnected support becomes "we can show that GANs can adapt to disconnected supports (under these other assumptions) that has not previously been shown for other methods." Not only is this a weaker result, but the degree to which you show GANs can take advantage of local smoothness is somewhat limited: at least with the parameter choices in Corollary 1, the smoothness only improves m dependence, not n dependence.
But in the GAN setting, m is essentially a question of how long you optimize for (and the relative rates between generator and discriminator updates, and various other questions like that out of scope for this paper), not any kind of externally fixed limitation like n. It's perhaps not too surprising that you don't show that optimizing a WGAN is statistically easier than estimating Wasserstein the distance, but given that estimating Wasserstein is so hard in high dimensions, it's a little disappointing. (Maybe it's easier, so that kappa is smaller, with locally smooth densities as here, though I don't know of any results like that offhand.)
(It's interesting, then, that in Corollary 1 the generator complexity depends only on m, basically the amount of optimization you're willing to do, while the discriminator complexity depends only on the available number of target samples n.)
Another concern is that to me, it is not very clear exactly what the statements of the assumptions mean. For example, does Theorem 1 apply only if I search over all generators in the class \mathcal G = \Xi(S_g, B_g, L_g)? In particular, does this mean that I have to consider all possible architectures matching those constraints, including allowing for all possible depths up to L_g, and all possible ways of allocating widths of the various interior layers / which weight entries are fixed at zero? It seems so, but this could be more explicit in the statements, even just by replacing "an existing \mathcal G" with "\mathcal G = \Xi(S_g, B_g, L_g)" in the statement of (e.g.) Theorem 1.
In your numerical experiments: you don't make it at all clear enough that you're plotting *different loss functions* for the GANs and the other methods! (You say this, but only in the text where it should definitely also be in the figure caption.) What happens if you plot the L2 difference for all methods, and the MMD/Wasserstein for all methods? Looking at Figure 6, it's not obvious that the GAN would do so substantially better. (It does seem to perhaps have the overall scale of the two components better than the other methods, but it doesn't look like as enormous a difference as it seems from Figure 5.)
Overall: Theorem 1 is of interest, but the results and especially the comparison to classic methods are not as resounding as they're presented here.
(Note that I have not (yet) verified or even really read most of the proofs; I might come back and do that later.)
Smaller points:
- I don't think that f-GAN actually fits in the framework (1) as you claim, since it needs to use the conjugate of f on the samples from Q. Also, the original GAN does fit into (1) but not your assumptions about the network form of f, since it needs a log in its activations.
- Another class of generative models where disconnected supports are really important is normalizing flows, which often build "bridges" between separated modes because (like your Lemma 2) their generators are constrained to be smooth and invertible. See e.g. Figure 2 of https://arxiv.org/abs/1810.01367 (who propose a new normalizing flow less susceptible to these problems).
- Remark 1 seems so obvious that it need not even be stated, since beta-smooth implies beta'-smooth for beta' < beta. It would only be interesting if you could actually take advantage of the smoother components somehow.
- Many papers in your bibliography are cited only as arXiv preprints when they were actually published in various places. For example, the first four papers were published at ICLR 2017, ICML 2017, ICML 2017, and ICLR 2018, respectively.
- There are many small typos and grammatical errors in the draft, including some that would be caught by a spell-checker ("methdos" on page 8), and an undefined LaTeX reference at the top of page 2. It would benefit from a thorough proofread.


LEARNING A SAT SOLVER FROM SINGLE-BIT SUPER- VISION
We present NeuroSAT, a message passing neural network that learns to solve SAT problems after only being trained as a classifier to predict satisfiability. Although it is not competitive with state-of-the-art SAT solvers, NeuroSAT can solve problems that are substantially larger and more difficult than it ever saw during training by simply running for more iterations. Moreover, NeuroSAT generalizes to novel distributions; after training only on random SAT problems, at test time it can solve SAT problems encoding graph coloring, clique detection, dominating set, and vertex cover problems, all on a range of distributions over small random graphs.

This paper presents the NeuroSAT architecture, which uses a deep, message passing neural net for predicting the satisfiability of CNF instances. The architecture is also able to predict a satisfiable assignment in the SAT case, and the literals involved in some minimal conflicting set of clauses (i.e. core) in the UNSAT case. The NeuroSAT architecture is based on a vector space embedding of literals and clauses, which exploits (with message passing) some important symmetries of SAT instances (permutation invariance and negation invariance). This architecture is tested on various classes of random SAT instances, involving both unstructured (RS) problems, and structured ones (e.g. graph colorings, vertex covers, dominating sets, etc.).
Overall the paper is well-motivated, and the experimental results are quite convincing. Arguably, the salient characteristic of NeuroSAT is to iteratively refine the confidence of literals voting for the SAT - or UNSAT - output, using a voting scheme on the last iteration of the literal matrix. This is very interesting, and NeuroSAT might be used to help existing solvers in choosing variable orderings for tackling hard instances, or hard queries (e.g. find a core).
On the other hand, the technical description of the architecture (sec. 3) is perhaps a little vague for having a clear intuition of how the classification task - for SAT instances - is handled in the NeuroSAT architecture. Namely, a brief description of the initial matrices (which encode the literal en clause embeddings) would be nice. Some comments on the role played by the multilayer perceptron units and the normalization units would also be welcome. The two update rules in Page 3 could be explained in more detail. For the sake of clarity, I would suggest to provide a figure for depicting a transition (from iteration t-1 to iteration t) in the architecture. As a minor comment, it would be nice (in Section 2) to define the main parameters $n$, $m$, and $d$ used in the rest of the paper.
Concerning the experimental part of the paper, Sections 4 & 5 are well-explained but, in Section 6,  the solution decoding method, inspired from PCA is a bit confusing. Specifically, we don’t know how a satisfying assignment is extracted from the last layer, and this should be explained in detail. According to Figure 4 and the comments above, it seems that a clustering method (with two centroids) is advocated, but this is not clear. In Table 1, the correlation between the accuracy on SAT instances, and the percent of SAT instances solved is not clear. Is the ratio of 70% measured on the CNF instances which have been predicted to be satisfiable? Or, is this ratio measured on the whole set of test instances? Finally, for the results established in Table 1, how many training instances and test instances have been used?
In Section 7, some important aspects related to experiments, are missing. In Sec 7.1, for SR(200) tasks, was NeuroSAT tested on the same conditions as those for SR(40) tasks? Notably, what is the input dimension $d$ of the embedding space here? (I guess that $d = 128$ is too small for such large instances). Also, how many training and test instances have been used to plot the curves in Figure 5? For the 4,888 satisfiable instances generated in Sec. 7.2, which solver have been used to determine the satisfiability of those instances (I guess it is Minisat, but this should be mentioned somewhere).
In Section 8, I found interesting the the ability of NeuroSAT in predicting the literals that participate in an UNSAT core. Indeed the problem of finding an UNSAT core in CNF instances is computationally harder than determining the satisfiability of this instance. So, NeuroSAT might be used here to help a solver in finding a core. But the notion of “confidence” should be explained in more detail in this section, and more generally, in the whole paper. Namely, it seems that in the last layer of each iteration, literals are voting for SAT (red colors) with some confidence (say $\delta$)  and voting for UNSAT (blue colors) with some confidence (say $\delta’$). Are $\delta$ and $\delta’$ correlated in the neural architecture? And, how confidences for UNSAT votes are updated?
Finally, I found that the different benchmarks where relevant, but I would also suggest (for future work, or in the appendix) to additionally perform experiments on the well-known random 3-SAT instances ($k$ is fixed to 3). Here, it is well-known that a phase transition (on the instances, not the solver/learner) occurs at 4.26 for the clause/variable ratio. A plot displaying the performance of NeuroSAT (accuracy in predicting the label of the instance) versus the clause/variable ratio would be very helpful in assessing the robustness of NeuroSAT on the so-called “hard” instances (which are close to 4.26). By extension, there have been a lot of recent work in generating “pseudo-industrial” random SAT instances, which incorporate some structure (e.g. communities) in order to mimic real-world structured SAT instances. To this point, it would be interesting to analyze the performance of NeuroSAT on such pseudo-industrial instances.


TRANSFERRING KNOWLEDGE ACROSS LEARNING PROCESSES
In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the final parameters of a source model will therefore struggle. Instead, transfer learning at a higher level of abstraction is needed. We propose Leap, a framework that achieves this by transferring knowledge across learning processes. We associate each task with a manifold on which the training process travels from initialization to final parameters and construct a meta-learning objective that minimizes the expected length of this path. Our framework leverages only information obtained during training and can be computed on the fly at negligible cost. We demonstrate that our framework outperforms competing methods, both in meta-learning and transfer learning, on a set of computer vision tasks. Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding reinforcement learning environments (Atari) that involve millions of gradient steps.

\documentclass[10pt]{article}
\usepackage{geometry}[1in]
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{enumerate}
\usepackage{indentfirst}
\begin{document}

	\section*{SUMMARY}

	The article proposes Leap, a novel meta-learning objective aimed at outperforming state-of-the-art approaches when dealing with collections of tasks that exhibit substantial between-task diversity.

	Similarly to prior work such as MAML [1] or Reptile [2], the goal of Leap is to learn an initialization $\theta_{0}$ for the model parameters, shared across tasks, which leads to good and data-efficient generalization performance when fine-tuning the model on a set of held-out tasks. In a nutshell, what sets Leap apart from MAML or Reptile is its cost function, which explicitly accounts for the entire path traversed by the model parameters during task-specific fine-tuning -- i.e., ``inner loop'' optimization --, rather than mainly focusing on the final value attained by the model parameters after fine-tuning. More precisely, Leap looks for an initialization $\theta_{0}$ of the model parameters such that the energy of the path traversed by $\gamma_{\tau}(\theta) = (\theta, f_{\tau}(\theta))$ while fine-tuning $\theta$ to optimize the loss $f_{\tau}(\theta)$ of a task $\tau$ is minimized, on average, across $\tau \sim p(\tau)$. Thus, it could be argued that Leap extends Reptile, which can be informally understood as seeking an initialization $\theta_{0}$ that minimizes the average squared Euclidean distance between $\theta_{0}$ and the model parameters after fine-tuning on each task $\tau \sim p(\tau)$ [2, Section 5.2], by using a distance function between initial and final model parameters that accounts for the geometry of the loss surface of each task during optimization.

	The final algorithm introduced in the paper considers however a variant of the aforementioned cost function, motivated by its authors on the basis of stabilising learning and eliminating the need for Hessian-vector products. The resulting approach is then evaluated on image recognition tasks (Omniglot plus a set of six additional computer vision datasets) as well as reinforcement learning tasks (Atari games).

	\section*{HIGH-LEVEL ASSESSMENT}

	The article proposes an interesting extension of existing work in meta-learning. In a slightly different context (meta-optimization), recent work [3] pointed out the existence of a ``short-horizon bias'' which could arise when using meta-learning objectives that apply only a small number of updates during ``inner-loop'' optimization. This observation is well-aligned with the motivation of this article, in which the authors attempt to complement successful methods like MAML or Reptile to perform well also in situations where a large number of gradient descent-based updates are applied during task-specific fine-tuning. Consequently, I believe the article is timely and relevant.

	Unfortunately, I have some concerns with the current version of the manuscript regarding (i) the proposed approach and the way it is motivated, (ii) the underlying theoretical results and, perhaps most importantly, (iii) the experimental evaluation. In my opinion, these should ideally be tackled prior to publication. Nonetheless, I believe that the proposed approach is promising and that these concerns can be either addressed or clarified. Thus I look forward to the rebuttal.

	\section*{MAJOR POINTS}

	\subsection*{1. Issues regarding proposed approach and its motivation/derivation}

	\textbf{1.a} Section 2.1 argues in favour of studying the path traversed by $\gamma_{\tau}(\theta) = (\theta, f_{\tau}(\theta))$ rather than the path traversed by the model parameters $\theta$ alone. However, this could in turn exacerbate the difficulty in dealing with collections of tasks for which the loss functions have highly diverse scales. For instance, taking the situation to the extreme, one could define an equivalence class of tasks $[\tau] = \left\{\tau \mid f_{\tau}(\theta) = g(\theta) + \mathrm{constant} \right\}$ such that any two tasks $\tau_{1}, \tau_{2} \in [\tau]$ would essentially represent the same underlying task, but could lead to arbitrarily different values of the Leap cost function.

	Given that Leap is a model-agnostic approach, like MAML or Reptile, and thus could be potentially applied in many different settings and domains, I believe the authors should study and discuss (theoretically or experimentally) the robustness of Leap with respect to between-task variation in the scale of the loss functions and, in case the method is indeed sensitive to those, propose an effective scheme to normalize them.

	\textbf{1.b} The current version of the manuscript motivates defining the cost function in terms of $\gamma_{\tau}(\theta) = (\theta, f_{\tau}(\theta))$ rather than the model parameters $\theta$ alone in order to ``avoid information loss'', making it seem that this modification is ``optional'' or, at least, not critical. Nevertheless, taking a closer look at the Leap objective and the meta-updates it induces, I believe it might actually be essential for the correctness of the approach. I elaborate this view in what follows. Let us write the Leap objective for a task $\tau$ as
	\[
	F_{\tau}(\theta_{0},\widetilde{\theta}_{0}) = \underbrace{\sum_{i=0}^{K_{\tau} - 1}{\left\vert\left\vert u^{(i+1)}_{\tau}(\widetilde{\theta}_{0}) - u^{(i)}_{\tau}(\theta_{0}) \right\vert\right\vert^{2}}}_{C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0})} + \underbrace{\sum_{i=0}^{K_{\tau} - 1}{\left( f_{\tau}\left(u^{(i+1)}_{\tau}(\widetilde{\theta}_{0})\right) - f_{\tau}\left(u^{(i)}_{\tau}(\theta_{0})\right) \right)^{2}}}_{C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})},
	\]
	where $\widetilde{\theta}_{0}$ denotes a ``frozen'' or ``detached'' copy of $\theta_{0}$ and $u^{(i)}_{\tau}$ maps $\theta_{0}$ to $\theta_{i}$, the model parameters after applying $i$ gradient descent updates to $f_{\tau}$ according to Equation (1) in the manuscript. Then, differentiating $C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0})$ and $C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})$ with respect to $\theta_{0}$ separately yields:
	\begin{align*}
	\nabla_{\theta_{0}} C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0}) &= -2 \sum_{i=0}^{K_{\tau} - 1}{J_{i}^{T}\left(\theta_{i+1} - \theta_{i} \right)} = -2 \alpha \sum_{i=0}^{K_{\tau} - 1}{J_{i}^{T} g_{i}} \\
	\nabla_{\theta_{0}} C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0}) &= -2 \sum_{i=0}^{K_{\tau} - 1}{\left(f_{\tau}(\theta_{i+1}) -  f_{\tau}(\theta_{i})\right) J_{i}^{T}g_{i}} = -2 \sum_{i=0}^{K_{\tau} - 1}{\Delta f^{i}_{\tau} J_{i}^{T}g_{i}}
	\end{align*}
	where $J_{i} = J_{\theta_{0}}u^{(i)}_{\tau}(\theta_{0})$ denotes the Jacobian of $u^{(i)}_{\tau}$ with respect to $\theta_{0}$, $g_{i} = \left. \nabla_{\theta} f_{\tau}(\theta)\right\rvert_{\theta=\theta_{i}}$ denotes the gradient of the loss function $f_{\tau}$ evaluated at $\theta_{i}$ and $\Delta f^{i}_{\tau} = f_{\tau}(\theta_{i+1}) -  f_{\tau}(\theta_{i})$ stands for the change in the loss function after the $i$-th update. To simplify the exposition, a constant ``inner-loop'' learning rate and no preconditioning were assumed, i.e., $\alpha_{i} = \alpha$ and $S_{i} = I$.

	Furthermore, the article claims that all Jacobian terms are approximated by identity matrices (i.e., $J_{i} = I$) as suggested in Section 5.2 of [1], leading to the following approximations:
	\begin{align*}
		\nabla_{\theta_{0}} C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0}) \approx -2 \alpha \sum_{i=0}^{K_{\tau} - 1}{ g_{i}} \\
		\nabla_{\theta_{0}} C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0}) \approx -2 \sum_{i=0}^{K_{\tau} - 1}{\Delta f^{i}_{\tau} g_{i}}
	\end{align*}

	Interestingly, it can be seen that the contribution to the meta-update of the energy of the path traversed by the model parameters $\theta$, $g_{\mathrm{Leap},1} =\nabla_{\theta_{0}} C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0})$, actually points in exactly the opposite direction than the meta-update of Reptile, given by $g_{\mathrm{Reptile}} = \sum_{i=0}^{K_{\tau} - 1}{g_{i}}$ (e.g. Equation (27) in [2]). In summary, if the Leap objective was defined in terms of $\theta$ rather than $(\theta, f_{\tau}(\theta))$, minimising the Leap cost function should maximise Reptile's cost function and viceversa. It is only the term $g_{\mathrm{Leap},2} =\nabla_{\theta_{0}} C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})$ that presumably ``re-aligns'' $g_{\mathrm{Reptile}}$ and $g_{\mathrm{Leap}} = g_{\mathrm{Leap},1} + g_{\mathrm{Leap},2}$. Indeed,
	\[
	g_{\mathrm{Leap}} = 2 \sum_{i=0}^{K_{\tau} - 1}{\left(-\Delta f^{i}_{\tau} - \alpha \right) g_{i}}
	\]
	will have positive inner product with $g_{\mathrm{Reptile}}$ if each gradient update yields a sufficient decrease in the loss $f_{\tau}$, that is, $\Delta f^{i}_{\tau} < -\alpha$.

	Moreover, I also wonder if this is the reason why the authors introduce the ``regularization'' term $\mu_{\tau}^{i}$, which as it currently stands in the manuscript, does not seem to relate in a particularly intuitive manner to the original objective of minimising the energy of $\gamma(t)$. By introducing $\mu_{\tau}^{i}$, the term $C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})$ becomes
	\[
		C^{\prime}_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0}) = \sum_{i=0}^{K_{\tau} - 1}{-\mathrm{sign}  \left( f_{\tau}\left(u^{(i+1)}_{\tau}(\widetilde{\theta}_{0})\right) - f_{\tau}\left(u^{(i)}_{\tau}(\theta_{0})\right) \right) \left( f_{\tau}\left(u^{(i+1)}_{\tau}(\widetilde{\theta}_{0})\right) - f_{\tau}\left(u^{(i)}_{\tau}(\theta_{0})\right) \right)^{2}},
	\]
	leading to $g^{\prime}_{\mathrm{Leap},2} = 2 \sum_{i=0}^{K_{\tau} - 1}{\vert \Delta f^{i}_{\tau} \vert g_{i}}$ and
	\[
	g^{\prime}_{\mathrm{Leap}} = 2 \sum_{i=0}^{K_{\tau} - 1}{\left(\vert \Delta f^{i}_{\tau} \vert - \alpha \right) g_{i}}.
	\]
	In turn, this relaxes the sufficient condition under which Leap and Reptile lead to meta-updates with positive inner product, namely, it changes the condition $\Delta f^{i}_{\tau} < -\alpha$ by a less restrictive counterpart $\vert \Delta f^{i}_{\tau} \vert \ge \alpha$.

	If these derivations happen to be correct, then I believe the way Leap is currently motivated in the article could be argued to be slightly misleading. What seems to be its main inspiration, accounting for the path that the model parameters traverse during fine-tuning, does not seem to be what drives the meta-updates towards the ``correct'' direction. Instead, the component of the objective due to the path traversed by the loss function values appears to be more important or, at least, not optional. Furthermore, I believe the regularization term $\mu_{\tau}^{i}$ should be better motivated, as the current version of the manuscript does not seem to justify its need clearly enough.

	Finally, under the assumption that the above is not mistaken, I wonder whether further tweaks to the meta-update, such as $g^{\prime\prime}_{\mathrm{Leap}} = 2 \sum_{i=0}^{K_{\tau} - 1}{\mathrm{max}\left(\vert \Delta f^{i}_{\tau} \vert - \alpha, 0 \right) g_{i}}$, could perhaps turn out to be helpful as well.
	\subsection*{2. Theoretical results}

	\textbf{2.a} Theorem 1 currently claims that the Pull-Forward algorithm converges to a local minimum of Equation (5). However, due to the non-convexity of the objective function, only convergence to a stationary point is established.

	\textbf{2.b} Most importantly, I am not entirely certain that the proof of Theorem 1 is complete in its current form. As I understand it, using the notation introduced by the authors in Appendix A, the following identities hold:
	\begin{align*}
		F(\psi_{s};\Psi_{s}) &= \mathbb{E}_{\tau,i} \vert\vert h_{\tau}^{i} - z_{\tau}^{i} \vert\vert^{2} \\
		F(\psi_{s+1};\Psi_{s}) &= \mathbb{E}_{\tau,i} \vert\vert h_{\tau}^{i} - x_{\tau}^{i} \vert\vert^{2} \\
		F(\psi_{s};\Psi_{s+1}) &= \mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - z_{\tau}^{i} \vert\vert^{2} \\
		F(\psi_{s+1};\Psi_{s+1}) &= \mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - x_{\tau}^{i} \vert\vert^{2}.
	\end{align*}

	The bulk of the proof is then devoted to show that $\mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - z_{\tau}^{i} \vert\vert^{2} = F(\psi_{s};\Psi_{s+1}) \ge \mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - x_{\tau}^{i} \vert\vert^{2} = F(\psi_{s+1};\Psi_{s+1})$. However, I do not immediately see how to make the final ``leap'' from $F(\psi_{s+1};\Psi_{s+1}) \le F(\psi_{s};\Psi_{s+1})$ to the actual claim of the Theorem, $F(\psi_{s+1};\Psi_{s+1}) \le F(\psi_{s};\Psi_{s})$.

	\subsection*{3. Experimental evaluation}

	\textbf{3.a} The experimental setup of Section 4.1 closely resembles experiments described in articles that introduced continual learning approaches, such as [4]. However, rather than including [4] as a baseline, the current manuscript compares against meta-learning approaches typically used for few-shot learning, such as MAML and Reptile. Consequently, I would argue the combination of experimental setup and selection of baselines is not entirely fair or, at least, it is incomplete.

	To this end, I would suggest to (i) include [4] (or a related continual learning approach) as an additional baseline in the experiments currently described in Section 4.1 as well as (ii) perform a new experiment to compare the performance of Leap to that of MAML and Reptile in few-shot classification tasks using OmniGlot and/or Mini-ImageNet as datasets.

	\textbf{3.b} The Multi-CV experiment described in Section 4.2 currently does not have strong baselines other than Leap. If possible, I would suggest including [5] in the comparison, as it is the article which inspired this particular experiment.

	\textbf{3.b} Likewise, the same holds for the experiment described in Section 4.3. In this case, I would suggest comparing to [4] for the same reason described above.

	\section*{MINOR POINTS}

	\begin{enumerate}

	\item In Section 2.1, it is claimed that "gradients that largely point in the same direction indicate a convex loss surface, whereas gradients with frequently opposing directions indicate an ill-conditioned loss landscape". Nevertheless, convex loss surfaces can in principle be ill-conditioned as well.

	\item Introducing a mathematical definition for the metric "area under the training curve" could make the experiment in Section 4.1 more self-contained.

	\item Several references are outdated, as they cite preprints that have since been accepted at peer-reviewed venues.

	\item The reinforcement learning experiments in Section 4.3 would benefit from additional runs with multiple seeds, and the subsequent inclusion of confidence intervals.

	\item I believe certain additional experiments could be insightful. For example, (i) studying how sensitive the performance of Leap is to parameter of the ``inner-loop'' optimizer (e.g. choice of
	optimizer, learning rate, batch size) or (ii) describing how the introduction of $\mu_{\tau}^{i}$ affects the performance of Leap.

	\end{enumerate}

	\section*{TYPOS}

	\begin{enumerate}

	\item The first sentence entirely in page 6 appears to have a superfluous word.

	\item The Taylor series expansion in the proof of Theorem 1 is missing the $O(\bullet)$ terms (or a $\approx$ sign).

	\item Also in the proof of Theorem 1, if $c_{\tau}^{i} = (\delta_{\tau}^{i})^{2} - \alpha_{\tau}^{i}\xi_{\tau}^{i}\delta_{\tau}^{i}$, wouldn't $\omega = \underset{\tau, i}{\mathrm{sup}} \langle \hat{x}^{i}_{\tau} - \hat{z}^{i}_{\tau}, g(\hat{x}^{i}_{\tau}) - g(\hat{z}^{i}_{\tau})\rangle + \xi_{\tau}^{i}\delta_{\tau}^{i}$ instead?

	\end{enumerate}
        \section*{ANSWER TO REBUTTAL}
        Please see comments in the thread.

	\section*{REFERENCES}

	\begin{enumerate}[ {[}1{]} ]
		\item Finn et al. ``Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.'' International Conference on Machine Learning. 2017.
		\item Nichol et al. ``On First-Order Meta-Learning Algorithms.'' arXiv preprint. 2018
		\item Wu et al. ``Understanding Short-Horizon Bias in Stochastic Meta-Optimization.'' International Conference on Learning Representations. 2018.
		\item Schwarz et al. ``Progress \& Compress: A scalable framework for continual learning.''  International Conference on Machine Learning. 2018.
		\item Serr{\`a} et al. ``Overcoming Catastrophic Forgetting with Hard Attention to the Task.''  International Conference on Machine Learning. 2018.
	\end{enumerate}
\end{document}


Current state-of-the-art methods for semantic segmentation use deep neural networks to learn the segmentation mask from the input image signal as an imageto-image mapping. While these methods effectively exploit global image context, the learning and computational complexities are high. We propose shared memory augmented neural network actors as a dynamically scalable alternative. Based on a decomposition of the image into a sequence of local patches, we train such actors to sequentially segment each patch. To further increase the robustness and better capture shape priors, an external memory module is shared between different actors, providing an implicit mechanism for image information exchange. Finally, the patch-wise predictions are aggregated to a complete segmentation mask. We demonstrate the benefits of the new paradigm on a challenging lung segmentation problem based on chest X-Ray images, as well as on two synthetic tasks based on the MNIST dataset. On the X-Ray data, our method achieves state-of-the-art accuracy with a significantly reduced model size compared to reference methods. In addition, we reduce the number of failure cases by at least half.

The authors applied the external memory module proposed by Graves et al. (2016) to the image segmentation task. SHAMANN is an extension to allow memory sharing between directions.
Authors claimed that one of the contributions is a reformulation of the semantic segmentation problem as a sequence learning task.
There are many previous works done in this direction,
- "Multi-Dimensional Recurrent Neural Networks", 2007
- "Scene Labeling with LSTM Recurrent Neural Networks", 2015
- "ReSeg: A Recurrent Neural Network-Based Model for Semantic Segmentation", 2016
- "Robust, Simple Page Segmentation Using Hybrid Convolutional MDLSTM Networks", 2017
and many more.
Authors should compare with those LSTM-based image segmentation approaches as well.
Their second contribution is a network with a shared external memory module between directions. However, the experiments are not enough to show the benefits of it. See the details below.
Handling long-range dependencies:
- In Section 3.3.2, authors mentioned that "One limitation of Bi-LSTM is that the number of network parameters grows proportionally to the memorization capacity, making it unsuitable for sequences with long-range dependencies.".
However, the experiments are not with long range sequences: 169 sequence length for X-ray dataset and 49 length for MNIST. A classic LSTM (not bi-directional) is known to handle up to 200 timesteps. Some comparison/analysis of handling long-range dependencies of Bi-LSTM, Bi-MANN, and SHAMANN are needed (ideally on high-resolution real images).
Dataset:
-  Authors compared 3 models only on MNIST. The structure on MNIST is simple, and the resolution of images is small to show the benefit of using (shared) external memory module instead of individual memory cells. It is not surprising that the reported performance difference is small. Authors could have reported such a comparison on X-ray dataset too but they did not. I would recommend authors pick another high-resolution real-image dataset and compare the performance of these 3 models.
Additional comparisons:
- Various patch size
- Longer sequence length
- Especially a trade-off between the patch size and the sequence length on the high resolution images (larger patch size with a shorter sequence length or shorter patch size with a longer sequence length)
- A comparison of Bi-LSTM with sharing weights will also be a good baseline.


ON THE UNIVERSAL APPROXIMABILITY AND COMPLEXITY BOUNDS OF QUANTIZED RELU NEURAL NETWORKS
Compression is a key step to deploy large neural networks on resource-constrained platforms. As a popular compression technique, quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight. In this paper, we study the representation power of quantized neural networks. First, we prove the universal approximability of quantized ReLU networks on a wide class of functions. Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and functiondependent structures. Our results reveal that, to attain an approximation error bound of , the number of weights needed by a quantized network is no more than O log 5 (1/ ) times that of an unquantized network. This overhead is of much lower order than the lower bound of the number of weights needed for the error bound, supporting the empirical success of various quantization techniques. To the best of our knowledge, this is the first in-depth study on the complexity bounds of quantized neural networks.

This paper studies the expressive power of quantized ReLU networks from a theoretical point of view. This is well-motivated by the recent success of using quantized neural networks as a compression technique. This paper considers both linear quantization and non-linear quantization, both function independent network structures and function dependent network structures. The obtained results show that the number of weights need by a quantized network is no more than polylog factors times that of a unquantized network. This justifies the use of quantized neural networks as a compression technique.
Overall, this paper is well-written and sheds light on a well-motivated problem, makes important progress in understanding the full power of quantized neural networks as a compression technique. I didn’t check all details of the proof, but the structure of the proof and several key constructions seem correct to me. I would recommend acceptance.
The presentation can be improved by having a formal definition of linear quantized networks and non-linear quantized networks, function-independent structure and function-dependent structure in Section 3 to make the discussion mathematically rigorous. Also, some of the ideas/constructions seem to follow (Yarotsky, 2017). It seems to be a good idea to have a paragraph in the introduction to have a more detailed comparison with (Yarotsky, 2017), highlighting the difference of the constructions, the difficulties that the authors overcame when deriving the bounds, etc.
Minor Comment: First paragraph of page 2: extra space after ``to prove the universal approximability’’.


In order to integrate uncertainty estimates into deep time-series modelling, Kalman Filters (KFs) (Kalman et al., 1960) have been integrated with deep learning models. Yet, such approaches typically rely on approximate inference techniques such as variational inference which makes learning more complex and often less scalable due to approximation errors. We propose a new deep approach to Kalman filtering which can be learned directly in an end-to-end manner using backpropagation without additional approximations. Our approach uses a highdimensional factorized latent state representation for which the Kalman updates simplify to scalar operations and thus avoids hard to backpropagate, computationally heavy and potentially unstable matrix inversions. Moreover, we use locally linear dynamic models to efficiently propagate the latent state to the next time step. While our locally linear modelling and factorization assumptions are in general not true for the original low-dimensional state space of the system, the network finds a high-dimensional latent space where these assumptions hold to perform efficient inference. This state representation is learned jointly with the transition and noise models. The resulting network architecture, which we call Recurrent Kalman Network (RKN), can be used for any time-series data, similar to a LSTM (Hochreiter and Schmidhuber, 1997) but uses an explicit representation of uncertainty. As shown by our experiments, the RKN obtains much more accurate uncertainty estimates than an LSTM or Gated Recurrent Units (GRUs) (Cho et al., 2014) while also showing a slightly improved prediction performance and outperforms various recent generative models on an image imputation task.

This paper proposes, Recurrent Kalman Network, a modified Kaman filter in which the latent dynamics is projected into a higher dimensional space; efficient inference in this high-dimensional latent space is possible due to the space being locally linear. The state representation, transition, and observation models are learned jointly by backpropagation.
The paper is well written and the model is clearly explained; I also like the simplicity of the idea that uses the same machinery of Kalman filter.  However, I believe the authors can improve the presentation of the model and empirical evaluation.
In terms of model presentation, the authors can compare the model with a large set of deep recurrent models that have recently been proposed for modeling time series with nonlinear latent dynamics (e.g. Variational Sequential Monte Carlo, Structured inference networks for nonlinear state space models, Black box variational inference for state space models, Composing graphical models with neural networks for structured representations and fast inference, etc.). For instance, a table of some of these models with their pros and cons can be helpful for guiding the reader.
In terms of model evaluation, the paper needs a better evaluation section specifically on the generative models (see examples above) that are much  more suitable for modeling uncertainty compared to LSTM/GRU. More specifically, another approach for alleviating the limitations of Kalman filter would be to use non-linear transitions based on some non-linear functions approximation. This approach has been proposed in deep Kaman filter (Krishnan 2015) and it would be interesting to see how well your model performs compared to that for modeling uncertainty and computing predictive log-likelihood.
In conclusion, I think the paper presents a nice idea but it requires more work in order to pass the ICLR acceptance threshold.
----------------------------------------------------------
The authors have addressed my comments and as a result I changed my rating to 6.


Recent advances in Neural Variational Inference allowed for a renaissance in latent variable models in a variety of domains involving high-dimensional data. In this paper, we introduce two generic Variational Inference frameworks for generative models of Knowledge Graphs; Latent Fact Model and Latent Information Model. While traditional variational methods derive an analytical approximation for the intractable distribution over the latent variables, here we construct an inference network conditioned on the symbolic representation of entities and relation types in the Knowledge Graph, to provide the variational distributions. The new framework can create models able to discover underlying probabilistic semantics for the symbolic representation by utilising distributions which permit training by back-propagation in the context of neural variational inference, resulting in a highly-scalable method. Under a Bernoulli sampling framework, we provide an alternative justification for commonly used techniques in large-scale stochastic variational inference, which drastically reduces training time at a cost of an additional approximation to the variational lower bound. The generative frameworks are flexible enough to allow training under any prior distribution that permits application of the re-parametrisation trick, as well as under any scoring function that permits maximum likelihood estimation of the parameters. Experimental results display the potential and efficiency of this framework by improving upon multiple benchmarks with Gaussian prior representations. Code publicly available on Github.

The paper presents two variational inference frameworks for generative models of knowledge graphs. Such models are based respectively on latent fact model and latent information model.
The authors argue that with the presented framework the underlying probabilistic semantics can be discovered. Experiments show performances comparable with state-of-art approaches.
Unfortunately, the paper seems to me not clear and rather incomplete in its actual form.
Overall, the proposal is novel. I cannot decide about significance because results do not outperform those of other approaches. To this extent, the authors should better discuss the results, explaining in more detail why this approach should be used instead of others (scales better, is faster, etc.).
In the abstract, it is asserted that one can discover underlying probabilistic semantics, but in the corpus of the paper this aspect is not described or mentioned in detail.
Similar problem for the reference to von-Mises distribution. This distribution is just named, it is said that the framework can handle such a distribution, but a reference to a paper and/or a short paragraph to explain the sentence are missing. This statement now results to be just information disconnected by the rest of the paper.
In a similar way, many other points suffer from a poor organization in my opinion.
When describing LIM an error is introduced here that is then copied and pasted throughout the paper: in the productory on p, p is in R not in E. This is a simple typo, but the fact that it is repeated so many times, also in the proof, gives me the feeling that the paper was written at the last moment.
Figures 1 and 2 are never referred.
Formula 6 must be better explained. If I have not lost something, n is the number of labeled triples, s_c is undefined, b_c is the probability of s_c to be equal to 1, the index i is never used. The paper lacks information here.
As regards the experimental part, some results are shown in subsection 4.3 called link prediction, others in section 5 called link prediction analysis. This organization does not seem to me to be really optimal. I would suggest creating an experimental section.
Moreover, the tests should be better explained, the tables are shown without specifying how they are built and how the values are collected. Information is provided in the appendix but could be included in the paper as the maximum limit is of 10 pages (8 suggested but I think an extra half page can be used).
The knowledge bases used should be at least cited, I know that freebase and wordnet are well-known but somewhere, in the description of the test, the name should be included. Also to specify the characteristics of the versions (WN18 vs WN18RR). Moreover, what does the value -257 in column 1, row 4 means?
Then, it is said that Table 1 shows improvements for ComplEx, but such improvements are rather low, is there a way to prove their significance? Otherwise, I would say that the performance is the same for WN18.
Tables 1 and 2 contain cells with '-' value, what does it mean?
Discussion about table 3 is incomplete in my opinion. First of all, the "proportion" column should be described. Also, on one hand, it is true that the _member_meronym is the least accurate and prominent but the most problem may come from _hypernym, which is the most prominent and the accuracy is also low. This fact is highlighted for table 4 but not for table 3.
Minor issues
- sec 3: references to Miao et al. must be enclosed by brackets
- sec 4.3: "We believe this *is* due to ..."
- sec 5.2: what is Model A? Also, the sentence seems incomplete.
Pros:
- Novel approach
Cons
- Test results are not convincing
- The paper is not mathematically sound
- The paper needs to be re-organized


The quest for increased visual recognition performance has led to the development of highly complex neural networks with very deep topologies. To avoid high computing resource requirements of such complex networks and to enable operation on devices with limited resources, this paper introduces adaptive kernels for convolutional layers. Motivated by the non-linear perception response in human visual cells, the input image is used to define the weights of a dynamic kernel called Adaptive kernel. This new adaptive kernel is used to perform a second convolution of the input image generating the output pixel. Adaptive kernels enable accurate recognition with lower memory requirements; This is accomplished through reducing the number of kernels and the number of layers needed in the typical CNN configuration, in addition to reducing the memory used, increasing 2X the training speed and the number of activation function evaluations. Our experiments show a reduction of 66X in the memory used for MNIST, maintaining 99% accuracy and 16X memory reduction for CIFAR10 with 92.5% accuracy.

The paper introduces adaptive kernels (that adapts its weights as a function of image content) to the framework of CNN. The benefit of adaptive kernels is the reduction of memory usage (at training and at the inference time) as well as training speedups (up to 2x). The kernels are evaluated on two datasets MNIST and CIFAR10
I like the idea of building models that are memory efficient at training and at evaluation time. However, the evaluation of the proposed adaptive kernels is rather limited. In order to improve the paper, the authors could take into consideration the following points:
1. Why there is still a need to combine adaptive convolutions with regular convolutions? What would the model performance be for a model with only adaptive kernels?
2. I might have missed it, but I couldn't find any motivation on why tanh is used as nonlinearity. Would the method work with relu?
3. Traditional convolutional kernels together with max pooling operations ensures some degree of translation invariance. How big is the generalization gap for the tested models when adaptive kernel is used?
4. How sensitive are the results to the number of adaptive kernels in the layers.
5. Adaptive kernels have only been tested in the first convolutional layer, would the adaptive kernels work well also in different layers?
6. On CIFAR10 the results seem to be worse that other methods. However, it is important to note that the Adaptive Kernels CNN has way less parameters. It would be interesting to see how the performance of adaptive kernels based CNNs scales with the number of parameters.
7. The evaluation on two datasets seems to be rather limited, additional comparisons should be included.
8. The authors acknowledge the similarities (and some differences) with Brabandere et al (2016). It might be beneficial to include comparison to this approach in the experimental section. Moreover, given the similarities, it might be good to discuss the differences in the approaches in the introduction section.
9. The ideas presented in the paper seems related to general concept of hypernetworks, where one network learns (or helps to learn) paramenters of the other network. It would be nice to position the ideas from the paper w.r.t. this line of research too.
10. Another related paper seems to be Spatial Transformer Networks (Jaderberg et al.).
I like the drawings, however, the font on the drawings is too small - making it hard to read.
Some typos:
1. the difficult to train the network
2. table 2: Dynamic -> Adaptive?
Overall, the paper presents interesting ideas with some degree of originality. I'd encourage the authors to extend the intro and position the ideas w.r.t. existing works and extend the evaluation.


Recently convolutional neural networks (CNNs) achieve great accuracy in visual recognition tasks. DenseNet becomes one of the most popular CNN models due to its effectiveness in feature-reuse. However, like other CNN models, DenseNets also face overfitting problem if not severer. Existing dropout method can be applied but not as effective due to the introduced nonlinear connections. In particular, the property of feature-reuse in DenseNet will be impeded, and the dropout effect will be weakened by the spatial correlation inside feature maps. To address these problems, we craft the design of a specialized dropout method from three aspects, dropout location, dropout granularity, and dropout probability. The insights attained here could potentially be applied as a general approach for boosting the accuracy of other CNN models with similar nonlinear connections. Experimental results show that DenseNets with our specialized dropout method yield better accuracy compared to vanilla DenseNet and state-of-the-art CNN models, and such accuracy boost increases with the model depth.

Overall Thoughts:
I think the use of regularisation to improve performance in DenseNet architectures is a topic of interest to the community. My concern with the paper in it’s current form is that the different dropout structures/schedules are priors and it is not clear from the current analysis exactly what prior is being specified and how to match that to a particular dataset. Further, I believe that the current presentation of the empirical results does not support the nature of the claims being made by the authors. I would be very interested to hear the authors’ comments on the following questions.
Specific Comments/Questions:
Sec1: Sorry if I have missed something but for the two reasons against std dropout on dense net, the reference supports the second claim but could a reference be provided to substantiate the first?
Sec1/2: The discussion around feature re-use needs to be clarified slightly in my opinion. Dropout can provide regularisation in a number of regimes - the term “feature reuse” is a little tricky because I can see the argument from both sides - under the authors arguments, forcing different features to be used can be a source or robustness so would not the level of granularity be something to be put in as a prior and not necessarily inherently correct or incorrect?
Sec3: The key contribution (in my opinion) suggested by the authors is the “detailed analysis” of their dropout structures. I’m afraid I didn’t see this in this section - there are a number of approaches that have been taken in the literature to analyse the regularisation properties of dropout - e.g. the insightful approach of Gal and Ghahramani on dropout as a Bayesian regulariser (as well as others). I was expecting to see something similar to this - could the authors comment on this? Would such an analysis be possible - it would reveal the true priors being applied by the different approaches and allow an analysis of the priors being applied by the different methods?
Sec3: Similarly, with the dropout probability schedules, there are practical methods for learning such probabilities during training (e.g. Concrete Dropout) - would it not be possible to learn these parameters with these approaches? Why do we need to set them according to fixed schedules? I think it would be necessary to demonstrate that a fixed schedule outperforms learned parameters.
Sec4: My main difficulty here is that the other key contribution of the paper are the claims constructed around empirical results. Throughout the results section, only single values are presented without attempt to measure the distributions of the results (not even error bars). Without this information it is impossible to make any statements on the significance of the results. Ideally histograms should be provided (rather than just error bars). How do we know the changes conferred are significant for the particular problems? How do we know that they are causal from the new structures and not from hyper parameters or optimisation effects?
Sec4: Dropout is the application of a prior - how do we know what this prior is doing and when it is sensible to apply it? How do we know the results will transfer to datasets other than CIFAR?
Sec4: Please could the authors provide justification to the claim that the improvements would increase with the depth of the network?
Refs: Please could the authors be sure to cite the published versions of articles (not ArXiv versions) when papers have been peer reviewed - e.g. the citation for DenseNet (among others)
Other Points:
Could the authors use text mode for sub or superscripts in maths equations when using words as opposed to symbols?
There are a number of uses of “could” when I don’t think the authors mean “could” - please could this be checked?
Typos:
p4 replying -> relying, whcih -> which


LEARNING FACTORIZED REPRESENTATIONS FOR OPEN-SET DOMAIN ADAPTATION
Domain adaptation for visual recognition has undergone great progress in the past few years. Nevertheless, most existing methods work in the so-called closed-set scenario, assuming that the classes depicted by the target images are exactly the same as those of the source domain. In this paper, we tackle the more challenging, yet more realistic case of open-set domain adaptation, where new, unknown classes can be present in the target data. While, in the unsupervised scenario, one cannot expect to be able to identify each specific new class, we aim to automatically detect which samples belong to these new classes and discard them from the recognition process. To this end, we rely on the intuition that the source and target samples depicting the known classes can be generated by a shared subspace, whereas the target samples from unknown classes come from a different, private subspace. We therefore introduce a framework that factorizes the data into shared and private parts, while encouraging the shared representation to be discriminative. Our experiments on standard benchmarks evidence that our approach outperforms the state of the art in open-set domain adaptation.

The paper addresses the problem of Domain Adaptation (DA) in an open setting (OSDA): while traditional DA assumes that the set of classes of the source and the target are identical, in Open-set DA, there are samples in the target which do not belong to any class in the source (unknown classes that I will outliers in this review). The main difficulty of Open-set DA is to simultaneously discard outliers and correctly classify other samples in the target. There are only two papers on Open-set DA so far, Busto'17 and Saito'18.
The method proposed by the authors can be summarized in a single equation, eq. 2, where they aim at learning a linear mapping to a latent space, which can be separated into two sub-spaces U (private space) and V (shared space) such that target outliers will be mapped to 0 in V while source and target non-outliers will be mapped to 0 in U, and hence separate outliers with non-outliers. To solve eq. 2, the authors convert it to Eqs. 3, 4, and 5 and apply techniques in Lee'07 and Mairal'14. The authors propose an extension for learning a linear classifier simultaneously and an extension for incorporating also unknown source classes (i.e. source outliers) when appropriate. An experimental evaluation on 2 datasets show the good performance of the method.
Pros:
-A novel method for a rather new and understudied so far, the work is then interesting for this setting
-Good results reported
Cons:
-The criterion used for choosing when examples are outliers seems heuristic, more discussion would be welcomed as well as some qualitative analysis for showing the interest of the method
-Existing baseline of Saito'18 not used in the 1st experiment
-Some parts require more justification
*Comments:
-The idea of the method is similar to the one of Jia'10 (Eq.6) for multi view learning, but this is rather new for Open-set DA.
-In order to separate target samples to either private or shared, the authors "encourage that either of these two parts (i.e. vectors T_i^u and T_i^v) goes to zero for each sample", which is reasonable. To achieve this the authors use sparse coding method coming from Lee'07. However, this does not make sense to me, because the sparse coding algorithm will encourage both T_i^u and T_i^v to be sparse, but nothing forces one of them to go to the zero-vector.
The authors should then better justify this choice. In particular, I wonder if adding explicitly the criterion used for identifying outliers as a new constraint to satisfy. Then, the optimization problem considered would make more sense to me.
Anyway, the authors could perform additional experiments to show the effectiveness of their method: (i) apply on a classic DA problem where we will expect that ||T^u|| or ||U|| (private subspace for outliers) should be close to zero.
Add a qualitative analysis on the values of  |T^u|| and |T^v|| - both in Open-set DA and classic DA - showing that the results are as expected.
- The 1st method (Eq.2) learns the latent space without using any label in the source (i.e there are only two labels: outlier or non-outlier, and all source samples are labeled non-outlier). Thus, the authors resort to the assumption that outliers are farther from source samples than non-outliers. This assumption is strong and may not hold in practice for two reasons: (1) the domain shift can be large and (2) without clustering techniques, many outliers can easily fall into the safe non-outliers zone (consider 0-4 for outliers and 5-9 for non-outliers, high chance this method will incorrectly classify 0 or 3 as non-outliers since 6,8,9 are already non-outliers).
- The Lagrange dual method (Lee'07, Eq. 6) solves an optimization problem with multiple quadratic constraints, i.e. ||U_j||^2 \le c for every j. However, the authors apply it to solve a problem (eq. 3 and 4) with a single linear constraint which is not quadratic: \sum ||U_j|| \le 1. Please explain:
(i) Why do you use that constrain instead of the one in Lee'07?
(ii) With your constrain, does the Lagrange dual method still work?
-The authors mention that they reported the results reported by Busto'17 in their experiment. Does this mean that the experiments were not reproduced? If so this seems rather unfair for other baselines since they may have worked on different instances.
Many baselines are not specific to Open-set DA, so it is rather expected to see bad results.
Since OSDA is new, it is true that there exists only two true baselines: Busto'17 and Saito'18. However, Saito'18 does not appear in BCIS benchmark (although appears in Office benchmark). Please add Saito'18 to the BCIS benchmark.
-The authors use fixed parameters for all the subproblems, I am a bit surprised by this choice, I would rather expect a parameterization task-dependent. Does this mean that the method is hard to tune ?
-The method seems complex, is there any convergence guarantee?
--
After rebuttal: thanks many points were answered.


OPPORTUNISTIC LEARNING: BUDGETED COST-SENSITIVE LEARNING FROM DATA STREAMS
In many real-world learning scenarios, features are only acquirable at a cost constrained under a budget. In this paper, we propose a novel approach for costsensitive feature acquisition at the prediction-time. The suggested method acquires features incrementally based on a context-aware feature-value function. We formulate the problem in the reinforcement learning paradigm, and introduce a reward function based on the utility of each feature. Specifically, MC dropout sampling is used to measure expected variations of the model uncertainty which is used as a feature-value function. Furthermore, we suggest sharing representations between the class predictor and value function estimator networks. The suggested approach is completely online and is readily applicable to stream learning setups. The solution is evaluated on three different datasets including the well-known MNIST dataset as a benchmark as well as two cost-sensitive datasets: Yahoo Learning to Rank and a dataset in the medical domain for diabetes classification. According to the results, the proposed method is able to efficiently acquire features and make accurate predictions.

This paper presents a novel method for budgeted cost sensitive learning from Data Streams.
This paper seems very similar to the work of Contrado’s RADIN algorithm which similarly evaluates sequential datapoints with a recurrent neural network by adaptively “purchasing” the most valuable features for the current datapoint under evaluation according to a budget.
In this process, a sample (S_i) with up to “d” features arrives for evaluation.  A partially revealed feature vector x_i arrives at time “t” for consideration.  There seems to exist a set of “known features” that that are revealed “for free” before the budget is considered (Algorithm 1).  Then while either the budget is not exhausted or some other stopping condition is met features are sequentially revealed either randomly (an explore option with a decaying rate of probability) or according to their cost sensitive utility.  When the stopping condition is reached, a prediction is made.  After a prediction is made, a random mini-batch of the partially revealed features is pushed into replay memory along with the correct class label and the P. Q, and target Q networks are updated.
The ideas of using a sequentially revealed vector of features and sequentially training a network are in Contrado’s RADIN paper.   The main novelty of the paper seems to be the use of MC dropout as an estimate of certainty in place of the softmax output layer and the methods of updating the P and Q networks.
The value of this paper is in the idea that we can learn online and in a cost sensitive way.  The most compelling example of this is the idea that a patient shows up at time “t” and we would like to make a prediction of disease in a cost sensitive way.  To this end I would have liked to have seen a chart on how well this algorithm performs across time/history.  How well does the algorithm perform on the first 100 patients vs the last 91,962-91,062 patients at what point would it make sense to start to use the algorithm (how much history is needed).
Am I correct in assuming there are some base features that are revealed “for free” for all samples?  If so how are these chosen?  If so how does the number of these impact the results?
In Contrado’s RADIN paper the authors explore both the MNIST dataset and others, including a medical dataset “cardio.”  Why did you only use RADIN as a comparison for the MNIST dataset and not the LTRC or diabetes dataset?  Did you actually re-implement RADIN or just take the numbers from their paper?  In which case, are you certain which MNIST set was used in this paper? (it was not as well specified as in your paper).
With respect to the real world validity of the paper, given that the primary value of the paper has to do with cost sensitive online learning, it would have been better to talk more about the various cost structure and how those impact the value of your algorithm.  For the first example, MNIST, the assumed uniform cost structure is a toy example that equates feature acquisition with cost.  The second example uses computational cost vs relevance gain.  This would just me a measure of computational efficiency, in which case all of the computational cost of running the updates to your networks should also be considered as cost.  With respect to the third proprietary diabetes dataset, the costs are real and relevant, however there discussion of these are given except to say that you had a single person familiar with medical billing create them for you (also the web address you cite is a general address and does not go to the dataset you are using).
 In reality, these costs would be bundled.  You say you estimate the cost in terms of overall financial burden, patient privacy and patient inconvenience.  Usually if you ask the patient to fill out a survey it has multiple questions, so for the same cost you get all the answers.  Similarly if you do a blood draw and test for multiple factors the cost to the patient and the hospital are paid for the most part upfront.  It is not realistic to say that the cost of asking a patient a questions is 1/20th of the cost of the survey.  The first survey question asked would be more likely 90-95% of the cost with each additional question some incremental percentage.  To show the value of your work, a better discussion of the cost savings would be appreciated.


In health, machine learning is increasingly common, yet neural network embedding (representation) learning is arguably under-utilized for physiological signals. This inadequacy stands out in stark contrast to more traditional computer science domains, such as computer vision (CV), and natural language processing (NLP). For physiological signals, learning feature embeddings is a natural solution to data insufficiency caused by patient privacy concerns -rather than share data, researchers may share informative embedding models (i.e., representation models), which map patient data to an output embedding. Here, we present the PHASE (PHysiologicAl Signal Embeddings) framework, which consists of three components: i) learning neural network embeddings of physiological signals, ii) predicting outcomes based on the learned embedding, and iii) interpreting the prediction results by estimating feature attributions in the "stacked" models (i.e., feature embedding model followed by prediction model). PHASE is novel in three ways: 1) To our knowledge, PHASE is the first instance of transferal of neural networks to create physiological signal embeddings. 2) We present a tractable method to obtain feature attributions through stacked models. We prove that our stacked model attributions can approximate Shapley values -attributions known to have desirable properties -for arbitrary sets of models. 3) PHASE was extensively tested in a cross-hospital setting including publicly available data. In our experiments, we show that PHASE significantly outperforms alternative embeddingssuch as raw, exponential moving average/variance, and autoencoder -currently in use. Furthermore, we provide evidence that transferring neural network embedding/representation learners between distinct hospitals can yield performant embeddings and offer recommendations when transference is ineffective.

Summary of the paper:
This paper proposes PHASE, a framework to learn the embeddings for physiological signals from medical records, which can be used in downstream prediction tasks, possibly across domains (i.e. different patient distribution). The authors employ separate LSTMs for each signal channel that are trained to predicts the minimum value of the signal in the fixed future time window (5 minutes in this paper). After training the LSTMs, the learned signal embeddings are fed to gradient boosted trees for a specific prediction task (e.g. predicting whether hypoxemia will occur in 5 minutes). Once the LSTMs are trained, they can be re-used for another dataset; the LSMTs are fixed, and generate embeddings that are fed to a new trainable gradient boosted trees for performing a similar task. The authors also combine existing attribution methods (DeepSHAP and Independent TreeSHAP) to provide some explanation of PHASE. The authors use three different datasets to test PHASE's prediction performance, transferability of the embeddings, and interpretation.
Pros:
- The paper is well-motivated, well-organized and clearly written. The reading experience was smooth.
- Given the importance of physiological signals in ICU settings, transferable embeddings can be an important technique in practice
- As the authors claim, I am not aware of any notable prior work on transferable physiological signal embeddings. The authors tackle a relatively unexplored territory.
Issues:
- The authors claim PHASE learns signal embeddings that are transferable. However, the authors train the embeddings to predict the minimum value within the next five time steps, because the downstream tasks are all predicting whether a certain signal goes below some threshold ("hypo"xenia, "hypo"capnia, "hypo"tension). This means the authors designed the embedding learning process with a priori knowledge of the downstream tasks, which significantly weakens theirs claim that PHASE learns transferable embeddings. Word embeddings trained on Wikipedia, or ConvNets trained on ImageNet are not designed to be used in a specific type of downstream tasks. What PHASE demonstrates is basically that "hypo"xxxx predictions can be accurately made with pre-training the embeddings to predict a very relevant task.
- The authors claim that transferred PHASE embeddings significantly outperform EMA or Raw. But I wouldn't call 0.005-0.02 AP improvement "significant". Model 12 in Figure 3 shows better performance than model 2 and 4, but the gap is not that large.
- More importantly, the fact that model 10 and model 12 show similar performance is not very surprising. The two hospitals are in the same city, only miles away. Naturally the distribution of the patients would not be too different. Given this, claiming that PHASE embeddings are transferable does not have a strong ground.
- The claim for transferable embedding is further weakened by Figure 4. Model 1^p in Figure 4 clearly performs worse than Raw, which means embeddings learned from significantly different setting (hospital P) is actually making it harder for XGB than simply looking at raw signals. If PHASE was learning a robust embeddings, then the learned embeddings should at least not hurt the performance of XGB.
- Evaluating the interpretation of the model is weak. All the authors did was pick four examples and provide qualitative explanation. And they do not even describe whether this interpretation is from model 9 or 10. It would have been much better if at least one medical expert took a look at more than a few examples. In the current form, we cannot be sure if the model is using the SaO2 signal in a medically meaningful way. Also, if this is the interpretation of model 10 or 12, then we should look at the attributions for other signals as well.
- Lack of description on experiment setup. The authors do not describe how they pre-trained the LSTMs to obtain Min^h, Auto^h and Hypox^h, which significantly hurts reproducibility. Also I couldn't find any description regarding train/test splits or cross validations, or size of the LSTM cells.
- More description is necessary as to how Raw was used to train XGB. Was the entire sequence of 15 signals fed to XGB?
- Y-axis of Figure 5 is not on the same scale. This makes it hard to intuitively understand the change of SaO2.


Recurrent neural networks (RNNs) are difficult to train on sequence processing tasks, not only because input noise may be amplified through feedback, but also because any inaccuracy in the weights has similar consequences as input noise. We describe a method for denoising the hidden state during training to achieve more robust representations thereby improving generalization performance. Attractor dynamics are incorporated into the hidden state to 'clean up' representations at each step of a sequence. The attractor dynamics are trained through an auxillary denoising loss to recover previously experienced hidden states from noisy versions of those states. This state-denoised recurrent neural network (SDRNN) performs multiple steps of internal processing for each external sequence step. On a range of tasks, we show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN with attractor dynamics on the hidden state but without the auxillary loss. We argue that attractor dynamics-and corresponding connectivity constraints-are an essential component of the deep learning arsenal and should be invoked not only for recurrent networks but also for improving deep feedforward nets and intertask transfer.

In this paper the authors develop the clever idea to use attractor networks, inspired by Hopfield nets, to “denoise” a recurrent neural network.  The idea is that for every normal step of an RNN, one induces an additional "dimension" of recurrency in order to create attractor dynamics around that particular hidden state. The authors introduce their idea and run some basic experiments. This paper is well written and the idea is novel (to me) and worthy of exploration.  Unfortunately, the experiments are seriously lacking in my opinion, as I believe *the major focus* of those experiments should be comparisons to other denoising / regularization techniques.
MAJOR
The point is taken that RNNs are susceptible to noise due to iterated application of the function. In my experience, countering noise (in the sense of gaussian noise added) isn’t a huge problem in practice because there are many regularization methodologies to handle it. This leads me to the point that I think the experiments need to compare across a number of regularization techniques.  The paper is motivated by discussion of noise, “noise robustness is a highly desirable property in neural networks”, and the experiments show improved performance on smaller datasets, all of which speak to regularization. So I believe comparisons with regularization techniques are pretty important here.
MODERATE
There is some motivation at the beginning of this piece, in particular about language, and does not contain citations, but should.
“Training is in complete batches to avoid the noise of mini-batch training.”  Please explain, I guess this is not a type of noise that the method handles?
What about problems that require graded responses, which is likely anything requiring integration? For example,  what happens in the majority task if the inputs were switched to a non-discrete version, where one must hold analog numbers?
MINOR
Any discussion about the (presumably dramatic) increase in training time due to the attractor dynamics unrolling + additional batching due to noise vectors (if I understood correctly)?
What are your confidence intervals over?  Presumably, we’d like to get confidence over multiple network instantiations.
Pg 1. Articulated neural network?
QUESTIONS
Does using a the ‘c’ variable as a bias instead of an initial condition really matter?
How does supervised training via eqn (4) relate to the classic training of Hopfield nets? I assume not at all, but it would be useful to clarify?
What RNN architecture did you use in the Figure 5 simulations (tanh vanilla RNN or GRU?)


Recent work has demonstrated the lack of robustness of well-trained deep neural networks (DNNs) to adversarial examples. For example, visually indistinguishable perturbations, when mixed with an original image, can easily lead deep learning models to misclassifications. In light of a recent study on the mutual influence between robustness and accuracy over 18 different ImageNet models, this paper investigates how training data affect the accuracy and robustness of deep neural networks. We conduct extensive experiments on four different datasets, including CIFAR-10, MNIST, STL-10, and Tiny ImageNet, with several representative neural networks. Our results reveal previously unknown phenomena that exist between the size of training data and characteristics of the resulting models. In particular, besides confirming that the model accuracy improves as the amount of training data increases, we also observe that the model robustness improves initially, but there exists a turning point after which robustness starts to decrease. How and when such turning points occur vary for different neural networks and different datasets.

This paper empirically evaluates the effect of the training dataset size on accuracy and robustness against adversarial attacks. The methodology of the paper is generally easy to assess and the overall idea well communicated.
For the motivation example I assume the following assessment holds true. Several linear functions are sampled and compose S_1, S_2, and T. A single linear regression model is used to fit all the data, either S_1 or (S_1 and S_2). If that is the case the experiment is not clear to me since the single linear model can only fit the data mean, mean slope (a) and constant (mu). Since the joint dataset better captures the mean of T the error for the joint training should be lower indeed. However, to actually compare both values the same threshold theta should be used for both and not a percentage of their performance. I would argue that this very simple model does not provide any valuable insight into the problem due to its construction.
The experimental setup presented in Section 4 only considers examples which are classified correctly by all data subsets. However, it is crucial to also consider the mistakes of these subsequent sets. For example, the learned model for the most restrictive dataset is most likely not exposed to a complex decision boundary, therefore it will exhibit a much smoother prediction at the cost that it will simply classify many more examples as the target class. In this case using data perturbations is not even the problem since completely different examples might be classified wrongly. Although not entirely clear, it would be very useful to consider the nearest negative neighbor in the dataset in the embedding space of the classifier to capture this problem at least partially. In general if the test accuracy is lower the learned classifier exhibits less performance, thus, adversary examples, distorted examples are not the main issue since it simply makes mistakes on visually different examples. Therefore, the overall analysis should be much more focused on models which achieve the same test performance but use require less data to achieve this performance.


CONTEXT-ADAPTIVE ENTROPY MODEL FOR END-TO- END OPTIMIZED IMAGE COMPRESSION
We propose a context-adaptive entropy model for use in end-to-end optimized image compression. Our model exploits two types of contexts, bit-consuming contexts and bit-free contexts, distinguished based upon whether additional bit allocation is required. Based on these contexts, we allow the model to more accurately estimate the distribution of each latent representation with a more generalized form of the approximation models, which accordingly leads to an enhanced compression performance. Based on the experimental results, the proposed method outperforms the traditional image codecs, such as BPG and JPEG2000, as well as other previous artificial-neural-network (ANN) based approaches, in terms of the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) index. The test code is publicly available at https://github.com/JooyoungLeeETRI/CA_Entropy_Model.

Update:
I have updated my review to mention that we should accept this work as being concurrent with the two papers that are discussed below.
Original review:
This paper is very similar to two previously published papers (as pointed by David Minnen before the review period was opened):
"Learning a Code-Space Predictor by Exploiting Intra-Image-Dependencies" (Klopp et al.)  from BMVC 2018,
and
"Joint Autoregressive and Hierarchical Priors for Learned Image Compression" (Minnen et al.) from NIPS 2018.
The authors have already tried to address these similarities and have provided a list in their reply, and my summary of the differences is as follows (dear authors: please comment if I am misrepresenting what you said):
(1) the context model is slightly different
(2) parametric model for hyperprior vs non-parametric
(3) this point is highly debatable to be considered as a difference because the distinction between using noisy outputs vs quantized outputs is a very tiny detail (any any practitioner would probably try both and test which works better).
(4) this is not really a difference. The fact that you provide details about the method should be a default! I want all the papers I read to have enough details to be able to implement them.
(5+)  not relevant for the discussion here.
If the results were significantly different from previous work, these differences would indeed be interesting to discuss, but they didn't seem to change much vs. previously published work.
If the other papers didn't exist, this would be an excellent paper on its own. However, I think the overlap is definitely there and as you can see from the summary above, it's not really clear to me whether this should be an ICLR paper or not. I am on the fence because I would expect more from a paper to be accepted to this venue (i.e., more than an incremental update to an existing set of models, which have already been covered in two papers).


The checkerboard phenomenon is one of the well-known visual artifacts in the computer vision field. The origins and solutions of checkerboard artifacts in the pixel space have been studied for a long time, but their effects on the gradient space have rarely been investigated. In this paper, we revisit the checkerboard artifacts in the gradient space which turn out to be the weak point of a network architecture. We explore image-agnostic property of gradient checkerboard artifacts and propose a simple yet effective defense method by utilizing the artifacts. We introduce our defense module, dubbed Artificial Checkerboard Enhancer (ACE), which induces adversarial attacks on designated pixels. This enables the model to deflect attacks by shifting only a single pixel in the image with a remarkable defense rate. We provide extensive experiments to support the effectiveness of our work for various attack scenarios using the state-of-the-art attack methods. Furthermore, we show that ACE is even applicable to large-scale datasets including ImageNet dataset and can be easily transferred to various pretrained networks.

I have to emphasize first that this is not my area of expertise so I am going to review it as an outsider.
The authors argue that the checkerboard phenomenon can be exploited to make neural networks robust against adversarial attacks. They propose to enhance the checkerboard pattern by first adding a layer, called Artificial Checkerboard Enhancer (ACE), and then evading the attacks by zero-padding the image. The authors’ argument is that enhancing the checkerboard phenomenon will make attacks more targeted towards certain pixels, which can be evaded by shifting the image.
Overall, I think the paper is difficult to read and is not suitable for publication. In terms of clarity, the authors do not use precise terminology that would allow the reader to reproduce their work. They allude to vague statements. For example, they introduce two KEY terminologies that are repeatedly used throughout the paper but are not properly defined (see for instance the “definition” of “Gradient Overlap” in Appendix C).
In addition, in terms of the experiements, it certainly does not help to say that they were “reproduced by [the authors themselves]”. What does this mean?

In terms of originality, I agree with the first reviewer that the defense strategy seems to be easily breakable. The authors propose that they enhance the checkerboard phenomenon so that adversarial attacks become easier to implement by targeting individual pixels (the pixels in the checkerboard artifacts). Then, they pad the image with zero pixels to shift it to the right. I don’t understand how shifting the pixels would make it harder to attack (especially when the adversary knows the system).
It would be really appreciated if the authors elaborate on the following points to help me understand their contribution:
- The entire discussion about ACE in Section 4.1 is ad-hoc and not well-motivated. Why would ACE enhance the checkerboard patterm? Can you please explain why it works? This is not mentioned anywhere in the paper. The experiment in Section 4.2 helps a bit but it does not answer this question.
- What wouldn't an adversary remove the padded pixels before generating the attack? In defense strategies, it is often assumed that the adversary knows the system. Can you please explain why that is not possible in this setting?
-
In Figure 4, the axes are \bar i and \bar j in the main body, but they are x and y in the figure. Please use the same notation.