ML algorithms or models, especially deep neural networks (DNNs), have shown significant promise in several areas. However, recently researchers have demonstrated that ML algorithms, especially DNNs, are vulnerable to adversarial examples (slightly perturbed samples that cause mis-classification). Existence of adversarial examples has hindered deployment of ML algorithms in safety-critical sectors, such as security. Several defenses for adversarial examples exist in the literature. One of the important classes of defenses are manifold-based defenses, where a sample is "pulled back" into the data manifold before classifying. These defenses rely on the manifold assumption (data lie in a manifold of lower dimension than the input space). These defenses use a generative model to approximate the input distribution. This paper asks the following question: do the generative models used in manifold-based defenses need to be topology-aware? Our paper suggests the answer is yes. We provide theoretical and empirical evidence to support our claim.
I. Summary of the Paper
This paper studies robustness to adversarial examples from the
perspective of having 'topology-aware' generative models. Next
to some experiments on data sets with a manifold structure, the
main contribution of the paper is a tandem of theorems that state
the conditions under which models can recover the topology---or
the number of connected components---of a data set correctly,
thereby making them more robust to adversarial examples.
II. Summary of the Review
This paper provides a novel perspective on adversarial examples through
the lens of Riemannian Geometry and topology. I appreciate novel
research that employs topology-based methods, but at present, I cannot
fully endorse accepting the paper. Specifically, I see the following
issues:
- Missing clarity: while the appendix is very comprehensive, which
  I appreciate, the main text could be improved; some statements appear
  redundant, while others need to be re-formulated to build intuition
- This paper appears to span both theory and applications. I appreciate
  this attempt, knowing full well that this is no easy feat to
  accomplish. However, the main theoretical result on the number of
  connected components only applies to mixtures of Gaussian
  distributions, but the purported scope of the paper is the analysis of
  manifold-based defences in general. I would expect a more in-depth
  discussion of the limitations of the theorem. Can we expect this to
  generalise? Moreover, 'topology' is reduced to 'connected components'
  in this paper. While this is perfectly adequate in the sense of
  connected components being a particular concept from topology, I would
  expect this to be clarified much earlier in the paper. In addition,
  connected components are a very basic and coarse concept, so I am
  wondering to what extent it is sufficient to describe models purely
  based on that information.
- As a sort of corollary to the previous point, the experiments could be
  improved. I like the idea of employing known data sets with a simple
  manifold structure, but the setup is somewhat preliminary; I would
  prefer to see an analysis of border cases or limit cases in which the
  theorem _almost_ applies (or not); plus, a more in-depth analysis of
  stochastic effects during training: do _all_ models end up being
  robust if their number of connected components is sufficiently large?
  Is there a dependency between the number of connected components and
  vulnerabilities---are models with a very small number of connected
  components more vulnerable than models with a very larger number of
  connected components? The present experimental section is lacking this
  depth.
- The same statements apply to the INC example. I found this super
  instructive, but it is only _one_ case on _one_ manifold---I would
  like to see more details here; maybe some of the experiments in the
  appendix could be moved to the front? I have some suggestions for
  shortening the paper (see below).
Despite these issues, I think this paper can be a strong contribution if
properly revised; since I am positive that at least some of these
suggestions could be performed within a revision cycle, I want to be
upfront and state that I will definitely consider raising my score,
provided that my concerns are addressed appropriately!
I have to state that I am _not_ an expert in adversarial examples, but
an expert in topology-based methods; I consider this paper to belong to
the latter field given its theoretical contributions about 'recovering'
the correct density distribution.
III. Clarity
The paper provides an extensive background to Riemannian geometry, which
I appreciated as a reference. Nevertheless, there are improvements to
the main text that I would suggest:
- Please consider changing the title to 'On the need...'
- The manifold assumption is that data lie _on_ a manifold or _close to_
  a manifold whose intrinsic dimension is much lower than that of the
  ambient space. This is not stated in sufficient precision in the
  paper; please correct the usage on p. 1 and p. 2
- In terms of notation, why use $p_M$ to denote the density on the whole
  of $\mathds{R}^n$? I would expect $p_M$ to refer to the density on $M$
  rather than the density of the whole space.
- If $M$ is a disjoint union of manifolds, please consider using
  a '\cupdot' operator to make this more clear.
- If the pairwise manifolds are disjoint, how can the resulting data
  distribution still contain any ambiguities? I find this hard to
  harmonise with the statement in Section 3.4 about the existence of
  a classifier that separates the manifolds. Please clarify the meaning
  behind the term 'ambiguities' here.
- The '(R0)' requirement definition and the discussion in Section 3.2
  strike me as needlessly complex. Would it be possible to shorten this
  or move some content to the appendix? I think it would be sufficient
  to have Eq. 1 and mention how it could be solved.
- Section 3.3 could be shortened as well, if I am not mistaken; while it
  is good to know how such models look, the 'change of variable formula'
  is not used directly any more in the paper; I think it might be easier
  to write down a generic form of the density for each model.
- The salient points of Section 3.4 seem to be the projection point;
  maybe this could also be shortened somewhat in the interest of having
  more space for experiments. The relevant information of this section
  was to learn how projections work for different models, but it would
  be sufficient to keep Eq. 3 and discuss Eq. 4 in the appendix
- The 'topological differences' mentioned in Section 4 could be
  clarified: the paper talks about differences in connected components.
- The $\lambda$-density set appears to be a superlevel set, if I am not
  mistaken: the level set would be defined for a single threshold only,
  while this paper introduces the pre-image of an interval.
- I would expect the pre-image to be defined as $p^{-1}$, not $p_{-1}$;
  the latter strikes me as somewhat non-standard usage
- The statements preceding Definition 1 require some more intuition;
  what is the purpose of these assumptions?
- The notation for the Euclidean balls should be briefly introduced
  before Definition 1. I recognise this as a standard notation, but
  since this is the first appearance of the symbol, it should be
  mentioned at least briefly.
- Definition 1 could also be phrased more intuitively; additional
  sentence behind each definition would be useful, such as:
  $\delta_\lambda$ is the largest $\delta$ such that the full
  superlevel set is contained in a ball of radius $\delta_\lambda$.
  Figure 1 is already helpful in that regard; it should ideally precede
  the definition and/or be made larger to be more illustrative
- Maybe the results of Theorem 2 could already be stated earlier; it
  could probably be explained reasonably well when describing the number
  of connected components as a sort of 'baseline' topological complexity
  that a model has to satisfy.
  On a more abstract level, could it also be summarised as 'the
  inclusion of prior knowledge is a necessary condition for robustness'?
- I would not state that the main goals of the experiments are to
  'check the correctness of Theorem 2'---the proofs should be
  responsible for this! I would rather say that the main goals are to
  provide empirical evidence for the *relevance* or *applicability* of
  the Theorem.
- The paragraph on 'Latent vector distributions' contains the most
  relevant information, viz. the knowledge of what the paper considers
  to be a 'topology-aware' and 'topology-ignorant' model; this should be
  highlighted more; maybe the figures could be extended to contain
  information about $n_X$?
Overall, I like the idea of having one overarching question in a paper
that is subsequently answered or discussed under different aspects. I
very much commend the authors of the paper for choosing this sort of
writing style!
IV. Experimental setup
As mentioned above, the experiments require more depth. I would propose
adding more repetitions of the training process for different data sets
and analysing the impact of the 'parameters' used in the theorems.
In particular, the 'INC' experiments show great promise for multiple
repetitions. Why not choose more data sets and more staring positions
and visualise the trajectories of _multiple_ draws, as shown for
a single draw in Figure 4 a,b,c?
Also, please consider moving the additional experiments from the
appendix to the main paper.
More compelling examples would also be helpful. Why not generate data
sets consisting of more than two manifolds? At present, the largest
issue I see in this section is that the conceptual 'leap of faith'
between the theory and the applications is simply too large. Would it
not be possible to perform the same experiments on a simple digits data
set, say MNIST?
V. Minor issues
The paper is well-written overall. There are only a few typos and minor
style issues that I would recommend fixing:
- please check the usage of quotes; it should be ``pulled back'', not ''pulled back'' in LaTeX
- please check the usage of citations; if '\citet' is used, citations
  can be used as nouns directly (for example: 'in Pennec (1999)' instead
  of 'in (Pennec, 1999)'.
- etc.. --> etc.
- it approximates posterior distribution --> it approximates a posterior distribution
- for compound distribution --> for a compound distribution
- a relatively simpler distribution --> a simpler distribution
- near manifold --> near a/the manifold
- we simply the minimum --> we denote (?) the minimum
- satisfies the followings --> satisfies the following properties
- number of connected component --> number of connected components
- Experimental result --> Experimental results
VI. Update after the rebuttal
The authors managed to address the majority of my comments. Overall, I still would like to see a more detailed/in-depth experimental setup, but I realise that this not directly possible within the timeframe allotted during the rebuttal period. I am thus raising my score.