Answers.Body: <p>To complete the <a href="https://ai.stackexchange.com/a/11201/22176">first answer</a> that is rather graph oriented, I will write a little about deep learning on manifolds, which is quite general in terms of GDL thanks to the nature of manifolds. </p>
  
  <p>Note that the description of GDL through the explanation of what are DL on graphs and manifolds, in opposition to DL on euclidean domains, comes from the 2017 paper <a href="https://arxiv.org/abs/1611.08097" rel="noreferrer">Geometric deep learning: going beyond Euclidean data</a> (this paper is excellent at clarifying both the intuition and the mathematics of what I'm writing).</p>
  
  <h2>1. In case you don't know what a manifold is</h2>
  
  <p>As the previously cited paper puts it:</p>
  
  <blockquote>
    <p>Roughly, a manifold is a space that is locally Euclidean. One of the
    simplest examples is a spherical surface modeling our planet: around a
    point, it seems to be planar, which has led generations of people to
    believe in the flatness of the Earth. Formally speaking, a
    (differentiable) d-dimensional manifold X is a topological space where
    each point x has a neighborhood that is topologically equivalent
    (homeomorphic) to a d-dimensional Euclidean space, called the tangent
    space.</p>
  </blockquote>
  
  <p>Good other not-so-technical <a href="https://stats.stackexchange.com/a/289488/241224">explanation</a> on stats.stackexchange</p>
  
  <p>Other <a href="https://en.wikipedia.org/wiki/Manifold#Motivating_examples" rel="noreferrer">Wikipedia examples</a> to develop not too abstract understanding</p>
  
  <p>Very shortly put, it's an interesting mathematical set on which to work (different kinds exist, see papers at the end of this answer for DL related manifolds uses). By work, you can typically understand that you <em>constrain the neural net parameters to the manifold</em> you chose (e.g. training with parameters constrained on a hypersphere, among the <a href="https://arxiv.org/abs/1805.08308" rel="noreferrer">geomstats paper</a> examples).</p>
  
  <p>Your data can also be represented thanks to a practical manifold. For example, you can choose to work on images and videos by representing the samples using Symmetric Positive Definite (SPD) matrices (see this <a href="https://arxiv.org/abs/1605.06182" rel="noreferrer">paper</a>), the space of SPD matrices being a manifold itself.</p>
  
  <h2>2. Why bother learning on manifolds ?</h2>
  
  <p>Defining a clearer/better adapted set (understand that it's a sort of constraint!) on which to learn parameters and features can make it simpler to formally understand what your model is doing, and can lead to better results. I see it as a part of the effort of deep learning formalization. One could say <strong>you're looking for the best information geometry for your task</strong>, the one that best captures the desirable data distribution properties.To develop this intuition consider the <a href="https://www.kaggle.com/apapiu/manifold-learning-and-autoencoders" rel="noreferrer">solar system analogy for manifold learning of this Kaggle kernel</a>:</p>
  
  <blockquote>
    <p>Perhaps a good analogy here is that of a solar system: the surface of our planets are the manifolds we're interested in, one for each digit. Now say you're on the surface of the earth which is a 2-manifold and you start moving in a random direction (let's assume gravity doesn't exist and you can go through solid objects). If you don't understand the structure of earth you'll quickly find yourself in space or inside the earth. But if you instead move within the local earth (say spherical) coordinates you will stay on the surface and get to see all the cool stuff.</p>
  </blockquote>
  
  <p>This analogy reminds us of the spherical surface planet model from <a href="https://arxiv.org/abs/1611.08097" rel="noreferrer">Bronstein's paper</a> already quoted above. This paper also describes a typical case for which manifolds are interesting: where graphs (the other example of GDL/DL on non euclidean data) are better at handling data from social or sensor networks, manifolds are good at modeling 3D objects endowed with properties like color texture in computer vision.</p>
  
  <h2>3. Regarding deep neural networks on manifolds</h2>
  
  <p>I would advise reading the <a href="https://github.com/geomstats/geomstats" rel="noreferrer">geomstats</a> <a href="https://arxiv.org/abs/1805.08308" rel="noreferrer">associated paper</a>, which does a great job at showing what it is and how it can be used, along with example codes (e.g. MNIST on hyperspheres manifold example code <a href="https://github.com/geomstats/applications/blob/master/deep_learning/mnist_hypersphere.py" rel="noreferrer">here</a>). This library implements manifolds and associated metrics on Keras. The choice of metrics is essential to understand the point of working on manifolds: it's because you need to work on an adapted mathematical set (<em>ie</em> with the right properties) with an adapted distance definition (<em>so that the measure actually means something when considering the problem you're trying to solve</em>) that you switch to working on manifolds.</p>
  
  <p>If you want to dive in the details and examples of deep learning on manifolds here are some papers:</p>
  
  <ul>
  <li><a href="https://arxiv.org/abs/1608.04233" rel="noreferrer">A Riemannian Network for SPD Matrix Learning</a>: new backpropagation to learn SPD matrices on Riemannian manifolds</li>
  <li><a href="https://arxiv.org/abs/1711.06540" rel="noreferrer">Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds</a>: using SPD matrices to aggregate convolutional features</li>
  </ul>
  
  <h2>4. Why <em>Riemannian</em> manifolds ?</h2>
  
  <p><em>TL;DR: you need a metric to do machine learning</em> (otherwise, how could you evaluate how much you actually learned !)</p>
  
  <p>Still based on <a href="https://arxiv.org/abs/1611.08097" rel="noreferrer">Bronstein's paper</a>:</p>
  
  <blockquote>
    <p>On each tangent space, we define an inner product [...]. This inner
    product is called a Riemannian metric in differential geometry and
    allows performing local measurements of angles, distances, and
    volumes. A manifold equipped with a metric is called a Riemannian
    manifold.</p>
  </blockquote>
  
  <h2>5. What's the relation between a Riemannian manifold and a Euclidean space ?</h2>
  
  <p>Still based on <a href="https://arxiv.org/abs/1611.08097" rel="noreferrer">Bronstein's paper</a>:</p>
  
  <blockquote>
    <p>a Riemannian manifold can be realized as a subset of a Euclidean space
    (in which case it is said to be <em>embedded</em> in that space) by using the
    structure of the Euclidean space to induce a Riemannian metric.</p>
  </blockquote>
  
  <p>I leave the details to the paper, otherwise this answer will never end.</p>
  
  <h2>6. Answers to questions in comments</h2>
  
  <p>Will only answer once I think I've found a relatively well-argued answer, so won't answer everything at once.</p>
  
  <ul>
  <li>Isn't manifold learning just a way of dimensionality reduction? </li>
  </ul>
  
  <p>I don't think so, it isn't <em>just that</em>. I haven't seen any dimensional reduction constraint (yet ?) in the papers I've read (cf. geomstats again). </p>
  
  <p>In the <a href="https://github.com/geomstats/applications/blob/master/deep_learning/mnist_hypersphere.py" rel="noreferrer">hypersphere/MNIST geomstats code example</a>, you can see the chosen manifold dimension <code>hypersphere_dimension = 17</code>. Since we're working with MNIST data I guess this would mean a dimension reduction in this particular case. I admit I would need to check exactly what that dimension implies on the neural net architecture, I haven't discussed my understanding of this yet.</p>
  
  <h2>Disclaimer</h2>
  
  <p>I'm still developing a more rigorous mathematical understanding of manifolds, and shall update this post to make additional necessary clarifications: exactly what can be considered as a manifold in a traditional deep learning context, why do we use the word manifold when speaking about the hidden state of auto-encoders (see the previously cited Kaggle kernel that quotes Goodfellow's book on this). All of this if the perfectly clear answer doesn't show up here before !</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-15T10:21:35.180Z
Answers.Text: I didn't talk about manifolds because I am not very familiar with this concept, and I think that many CS/AI students will not be familiar with such concept. I think you should give an intuitive and brief explanation of the meaning of a manifold, in order to fill this gap. How do you intuitively think of a manifold? Isn't a manifold also "Euclidean data"? Why isn't e.g. an image a manifold? I think your answer would improve a lot if you explain and clarify these points. Btw, see this related question: https://ai.stackexchange.com/q/11226/2444. Feel free to answer it, if you are able to do it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-15T10:28:08.517Z
Answers.Text: I think you should also consider the fact that people are used to look at datasets as a design matrix or a bunch of feature vectors (for each observation). How would this compare to manifolds and non-Euclidean data? Maybe, if you are able, you should answer these questions on the linked and related question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-15T13:43:58.107Z
Answers.Text: Isn't manifold learning just a way of dimensionality reduction?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-18T09:05:17.867Z
Answers.Text: Manifold is easy to understand if your understand euclidean space. Euclidean space is a space of n-dimensional vectors of real numbers. Manifold is a space (set of) vectors that for any point  of it small area around it could be transformed (smoothly and reversibly) to ball of Euclidean space. So manifold is locally almost Euclidean, but globally generally is not. That mean that every small area of manifold can be approximated by hyperplane (the smaller area, the more precise), but globally it can not be. Manifolds are introduced in differential topology and differential geometry.
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-18T20:48:35.530Z
Answers.Text: @mirror2image You should have described what an Euclidean space is. What are the necessary properties that a space must have to be considered Euclidean? You should also explain what you mean by "locally" and "globally".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-19T06:07:45.120Z
Answers.Text: @nbro I purposefully didn't go into detail and tried explain intuition behind it. Formal definitions would be just copy-past from textbooks. Euclidean space is just  R^n which used in analysis and linear algebra. Locally and globally in depth - that require  some differential topology, where each explanation require more explanations until all the textbook is copied....
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-19T16:18:47.983Z
Answers.Text: I think your answer might be unnecessarily long and complex. For a long overview, I will just look at the paper I mentioned. I think you should try to compress your answer and just leave the key points/insights regarding GDL on manifolds. Note that this does not mean that you can't try to give an explanation of manifolds!
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-02T13:37:09.520Z
Answers.Text: @Bluepon: I'd be very very interested in working in geometric machine learning and currently looking for a postdoc in France. I've a PhD in differential geometry and also previous postdoctoral experience working in machine learning. Any idea if some professor does it in France? Thanks a lot!
Answers.UserDisplayName: Learning Math
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-15T09:15:07.867Z
Answers.LastActivityDate: 2019-03-19T08:41:01.907Z
Answers.LastEditDate: 2019-03-19T08:41:01.907Z
Answers.UserDisplayName: Blupon
Answers.Body: <p>I didn't read the paper in depth, but one example of where assumptions of Euclidean space are made in the design of the networks are with ConvNets in image processing.</p>
  <p>Specifically, Euclidean spaces are transformationally invariant, meaning that <span class="math-container">$d(a,b) = d(a+c,b+c)$</span>. Each convolution layer iterates over the image with a certain amount of stride, which guarantees a certain amount of translational invariance in exchange for a smaller size of network parameters.</p>
  <p>ConvNets would not naively work for a dataset which comes from, say, a graph, because its not clear how to stride over it, as a possible example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T16:22:37.780Z
Answers.Text: Maybe this would be the answer to [my other related question](https://ai.stackexchange.com/q/11226/2444)? But I've just noticed that you also provided an answer there.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T15:45:22.720Z
Answers.LastActivityDate: 2021-08-25T15:45:22.720Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>The article <a href="https://arxiv.org/abs/1611.08097" rel="nofollow noreferrer">Geometric deep learning: going beyond Euclidean data</a> (by Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, Pierre Vandergheynst) provides an overview of this relatively new sub-field of deep learning. It answers all the questions asked above (and more). If you are familiar with deep learning, graphs, linear algebra and calculus, you should be able to follow this article.</p>
  <p><em>What is geometric deep learning (GDL)</em>?</p>
  <p>This article describes GDL as follows</p>
  <blockquote>
  <p>Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds.</p>
  </blockquote>
  <p>So, the inputs to these GDL models are graphs (or representations of graphs), or, in general, any <a href="https://ai.stackexchange.com/q/11226/2444">non-Euclidean data</a>. To be more concrete, the input to these models (e.g. <a href="https://ai.stackexchange.com/q/11169/2444">graph neural networks</a>) are e.g. feature vectors associated with the nodes of the graphs and matrices which describe the graph structure (e.g. the <a href="https://en.wikipedia.org/wiki/Adjacency_matrix" rel="nofollow noreferrer">adjacency matrix</a> of the graphs).</p>
  <p><em>Why are e.g. graphs non-Euclidean data?</em></p>
  <p>A graph is a non-Euclidean structure because e.g. distances between nodes are not well defined. Yes, you can have graphs with weights associated with the edges, but not all graphs have this property.</p>
  <p><em>What classes of problems does GDL address?</em></p>
  <p>In GDL, there are two classes of problems that are often tackled:</p>
  <ol>
  <li>characterise the structure of the data (e.g. of a graph)</li>
  <li>analyse functions defined on a given non-Euclidean domain</li>
  </ol>
  <p>These classes of problems are related, given that the structure of the graph imposes certain properties on the functions that can be defined on it. Furthermore, these properties of these functions can also convey information about the structure of the graph.</p>
  <p><em>What are applications of GDL?</em></p>
  <p>An example of an application where this type of data (graphs) arises is in the context of social networks, where each user can be associated with a vertex of the social graph and the characteristics (or features) of each user (e.g. number of friends) can be represented as a feature vector (which can then be associated with the corresponding vertex of a graph). In this context, the goal might e.g. be to determine different groups of users in the social network (i.e. clustering).</p>
  <p><em>Why can't we simply use deep learning methods (like CNNs) when the data is non-Euclidean?</em></p>
  <p>There are several problems that arise when dealing with non-Euclidean data. For example, operations like convolution are not (usually) defined on non-Euclidean data. More concretely, the relative position of nodes is not defined on graphs (but this would be required to perform the usual convolution operation): in other words, it is meaningless to talk about a vertex that is e.g. on the left of another vertex. In practice, it means that we can't simply use the usual CNN when we are given non-Euclidean data. There have been attempts to generalise the convolution operation to graphs (or to approximate it). The field is still quite new, so there will certainly be new developments and breakthroughs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-14T09:36:51.310Z
Answers.Text: I need to improve this answer by explaining more in detail the meaning of "characterize the structure of the data" and "analyze functions defined on a given non-Euclidean domain". I should also explain what the output of these GDLs models are and how can they be trained.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T16:11:53.510Z
Answers.Text: I don't like this answer that I gave. I may come back to it and rewrite it completely.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T08:50:30.870Z
Answers.Text: I'm not clear what you mean when you say  the distance between two nodes is not well defined(graphs are non euclidean data). I'm not sure, but I believe it indicates that if we want to measure the distance between two nodes, we can't just draw a straight line and measure the distance; instead, we must walk through the edges and count the number of edges to get an accurate distance measurement. As a result, graphs aren't euclidean.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T17:39:23.130Z
Answers.LastActivityDate: 2021-02-03T01:27:19.697Z
Answers.LastEditDate: 2021-02-03T01:27:19.697Z
Answers.UserDisplayName: nbro
Body: <p>What is geometric deep learning (GDL)?</p>
  <p>Here are a few sub-questions</p>
  <ul>
  <li>How is it different from deep learning?</li>
  <li>Why do we need GDL?</li>
  <li>What are some applications of GDL?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-28T09:02:15.760Z
Comments.Text: you can check this https://www.youtube.com/playlist?list=PLn2-dEmQeTfQ8YVuHBOvAhUlnIPYxkeu3 this link for a deep understanding about GDL.
Comments.UserDisplayName: Swakshar Deb
Tags: <deep-learning><definitions><geometric-deep-learning><graphs><graph-neural-networks>
Title: What is geometric deep learning?
UserDisplayName: nbro
ThreadText: # What is geometric deep learning?
  
  (Asked by: username_0 on 2019-03-12)
  
  What is geometric deep learning (GDL)?
  
  Here are a few sub-questions
  
  * How is it different from deep learning?
  * Why do we need GDL?
  * What are some applications of GDL?
  
  **Tags:** deep-learning, definitions, geometric-deep-learning, graphs, graph-neural-networks
  > **Comments:**
  > **username_1**: you can check this https://www.youtube.com/playlist?list=PLn2-dEmQeTfQ8YVuHBOvAhUlnIPYxkeu3 this link for a deep understanding about GDL.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2019-03-13. Score: 8 | Accepted answer
  
  The article [Geometric deep learning: going beyond Euclidean data](https://arxiv.org/abs/1611.08097) (by Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, Pierre Vandergheynst) provides an overview of this relatively new sub-field of deep learning. It answers all the questions asked above (and more). If you are familiar with deep learning, graphs, linear algebra and calculus, you should be able to follow this article.
  
  *What is geometric deep learning (GDL)*?
  
  This article describes GDL as follows
  
  > Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds.
  
  So, the inputs to these GDL models are graphs (or representations of graphs), or, in general, any [non-Euclidean data](https://ai.stackexchange.com/q/11226/2444). To be more concrete, the input to these models (e.g. [graph neural networks](https://ai.stackexchange.com/q/11169/2444)) are e.g. feature vectors associated with the nodes of the graphs and matrices which describe the graph structure (e.g. the [adjacency matrix](https://en.wikipedia.org/wiki/Adjacency_matrix) of the graphs).
  
  *Why are e.g. graphs non-Euclidean data?*
  
  A graph is a non-Euclidean structure because e.g. distances between nodes are not well defined. Yes, you can have graphs with weights associated with the edges, but not all graphs have this property.
  
  *What classes of problems does GDL address?*
  
  In GDL, there are two classes of problems that are often tackled:
  
  1. characterise the structure of the data (e.g. of a graph)
  2. analyse functions defined on a given non-Euclidean domain
  
  These classes of problems are related, given that the structure of the graph imposes certain properties on the functions that can be defined on it. Furthermore, these properties of these functions can also convey information about the structure of the graph.
  
  *What are applications of GDL?*
  
  An example of an application where this type of data (graphs) arises is in the context of social networks, where each user can be associated with a vertex of the social graph and the characteristics (or features) of each user (e.g. number of friends) can be represented as a feature vector (which can then be associated with the corresponding vertex of a graph). In this context, the goal might e.g. be to determine different groups of users in the social network (i.e. clustering).
  
  *Why can't we simply use deep learning methods (like CNNs) when the data is non-Euclidean?*
  
  There are several problems that arise when dealing with non-Euclidean data. For example, operations like convolution are not (usually) defined on non-Euclidean data. More concretely, the relative position of nodes is not defined on graphs (but this would be required to perform the usual convolution operation): in other words, it is meaningless to talk about a vertex that is e.g. on the left of another vertex. In practice, it means that we can't simply use the usual CNN when we are given non-Euclidean data. There have been attempts to generalise the convolution operation to graphs (or to approximate it). The field is still quite new, so there will certainly be new developments and breakthroughs.
  
  > **Comments:**
  > **username_0**: I need to improve this answer by explaining more in detail the meaning of "characterize the structure of the data" and "analyze functions defined on a given non-Euclidean domain". I should also explain what the output of these GDLs models are and how can they be trained.  
  > **username_0**: I don't like this answer that I gave. I may come back to it and rewrite it completely.  
  > **username_1**: I'm not clear what you mean when you say  the distance between two nodes is not well defined(graphs are non euclidean data). I'm not sure, but I believe it indicates that if we want to measure the distance between two nodes, we can't just draw a straight line and measure the distance; instead, we must walk through the edges and count the number of edges to get an accurate distance measurement. As a result, graphs aren't euclidean.  
  
  ---
  
  ### Answer by username_2 on 2019-03-15. Score: 20
  
  To complete the [first answer](https://ai.stackexchange.com/a/11201/22176) that is rather graph oriented, I will write a little about deep learning on manifolds, which is quite general in terms of GDL thanks to the nature of manifolds.
  
  Note that the description of GDL through the explanation of what are DL on graphs and manifolds, in opposition to DL on euclidean domains, comes from the 2017 paper [Geometric deep learning: going beyond Euclidean data](https://arxiv.org/abs/1611.08097) (this paper is excellent at clarifying both the intuition and the mathematics of what I'm writing).
  
  ## 1. In case you don't know what a manifold is
  
  As the previously cited paper puts it:
  
  > Roughly, a manifold is a space that is locally Euclidean. One of the
  > simplest examples is a spherical surface modeling our planet: around a
  > point, it seems to be planar, which has led generations of people to
  > believe in the flatness of the Earth. Formally speaking, a
  > (differentiable) d-dimensional manifold X is a topological space where
  > each point x has a neighborhood that is topologically equivalent
  > (homeomorphic) to a d-dimensional Euclidean space, called the tangent
  > space.
  
  Good other not-so-technical [explanation](https://stats.stackexchange.com/a/289488/241224) on stats.stackexchange
  
  Other [Wikipedia examples](https://en.wikipedia.org/wiki/Manifold#Motivating_examples) to develop not too abstract understanding
  
  Very shortly put, it's an interesting mathematical set on which to work (different kinds exist, see papers at the end of this answer for DL related manifolds uses). By work, you can typically understand that you *constrain the neural net parameters to the manifold* you chose (e.g. training with parameters constrained on a hypersphere, among the [geomstats paper](https://arxiv.org/abs/1805.08308) examples).
  
  Your data can also be represented thanks to a practical manifold. For example, you can choose to work on images and videos by representing the samples using Symmetric Positive Definite (SPD) matrices (see this [paper](https://arxiv.org/abs/1605.06182)), the space of SPD matrices being a manifold itself.
  
  ## 2. Why bother learning on manifolds ?
  
  Defining a clearer/better adapted set (understand that it's a sort of constraint!) on which to learn parameters and features can make it simpler to formally understand what your model is doing, and can lead to better results. I see it as a part of the effort of deep learning formalization. One could say **you're looking for the best information geometry for your task**, the one that best captures the desirable data distribution properties.To develop this intuition consider the [solar system analogy for manifold learning of this Kaggle kernel](https://www.kaggle.com/apapiu/manifold-learning-and-autoencoders):
  
  > Perhaps a good analogy here is that of a solar system: the surface of our planets are the manifolds we're interested in, one for each digit. Now say you're on the surface of the earth which is a 2-manifold and you start moving in a random direction (let's assume gravity doesn't exist and you can go through solid objects). If you don't understand the structure of earth you'll quickly find yourself in space or inside the earth. But if you instead move within the local earth (say spherical) coordinates you will stay on the surface and get to see all the cool stuff.
  
  This analogy reminds us of the spherical surface planet model from [Bronstein's paper](https://arxiv.org/abs/1611.08097) already quoted above. This paper also describes a typical case for which manifolds are interesting: where graphs (the other example of GDL/DL on non euclidean data) are better at handling data from social or sensor networks, manifolds are good at modeling 3D objects endowed with properties like color texture in computer vision.
  
  ## 3. Regarding deep neural networks on manifolds
  
  I would advise reading the [geomstats](https://github.com/geomstats/geomstats) [associated paper](https://arxiv.org/abs/1805.08308), which does a great job at showing what it is and how it can be used, along with example codes (e.g. MNIST on hyperspheres manifold example code [here](https://github.com/geomstats/applications/blob/master/deep_learning/mnist_hypersphere.py)). This library implements manifolds and associated metrics on Keras. The choice of metrics is essential to understand the point of working on manifolds: it's because you need to work on an adapted mathematical set (*ie* with the right properties) with an adapted distance definition (*so that the measure actually means something when considering the problem you're trying to solve*) that you switch to working on manifolds.
  
  If you want to dive in the details and examples of deep learning on manifolds here are some papers:
  
  * [A Riemannian Network for SPD Matrix Learning](https://arxiv.org/abs/1608.04233): new backpropagation to learn SPD matrices on Riemannian manifolds
  * [Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds](https://arxiv.org/abs/1711.06540): using SPD matrices to aggregate convolutional features
  
  ## 4. Why *Riemannian* manifolds ?
  
  *TL;DR: you need a metric to do machine learning* (otherwise, how could you evaluate how much you actually learned !)
  
  Still based on [Bronstein's paper](https://arxiv.org/abs/1611.08097):
  
  > On each tangent space, we define an inner product [...]. This inner
  > product is called a Riemannian metric in differential geometry and
  > allows performing local measurements of angles, distances, and
  > volumes. A manifold equipped with a metric is called a Riemannian
  > manifold.
  
  ## 5. What's the relation between a Riemannian manifold and a Euclidean space ?
  
  Still based on [Bronstein's paper](https://arxiv.org/abs/1611.08097):
  
  > a Riemannian manifold can be realized as a subset of a Euclidean space
  > (in which case it is said to be *embedded* in that space) by using the
  > structure of the Euclidean space to induce a Riemannian metric.
  
  I leave the details to the paper, otherwise this answer will never end.
  
  ## 6. Answers to questions in comments
  
  Will only answer once I think I've found a relatively well-argued answer, so won't answer everything at once.
  
  * Isn't manifold learning just a way of dimensionality reduction?
  
  I don't think so, it isn't *just that*. I haven't seen any dimensional reduction constraint (yet ?) in the papers I've read (cf. geomstats again).
  
  In the [hypersphere/MNIST geomstats code example](https://github.com/geomstats/applications/blob/master/deep_learning/mnist_hypersphere.py), you can see the chosen manifold dimension `hypersphere_dimension = 17`. Since we're working with MNIST data I guess this would mean a dimension reduction in this particular case. I admit I would need to check exactly what that dimension implies on the neural net architecture, I haven't discussed my understanding of this yet.
  
  ## Disclaimer
  
  I'm still developing a more rigorous mathematical understanding of manifolds, and shall update this post to make additional necessary clarifications: exactly what can be considered as a manifold in a traditional deep learning context, why do we use the word manifold when speaking about the hidden state of auto-encoders (see the previously cited Kaggle kernel that quotes Goodfellow's book on this). All of this if the perfectly clear answer doesn't show up here before !
  
  > **Comments:**
  > **username_0**: I didn't talk about manifolds because I am not very familiar with this concept, and I think that many CS/AI students will not be familiar with such concept. I think you should give an intuitive and brief explanation of the meaning of a manifold, in order to fill this gap. How do you intuitively think of a manifold? Isn't a manifold also "Euclidean data"? Why isn't e.g. an image a manifold? I think your answer would improve a lot if you explain and clarify these points. Btw, see this related question: https://ai.stackexchange.com/q/11226/2444. Feel free to answer it, if you are able to do it.  
  > **username_0**: I think you should also consider the fact that people are used to look at datasets as a design matrix or a bunch of feature vectors (for each observation). How would this compare to manifolds and non-Euclidean data? Maybe, if you are able, you should answer these questions on the linked and related question.  
  > **username_0**: Isn't manifold learning just a way of dimensionality reduction?  
  > **username_3**: Manifold is easy to understand if your understand euclidean space. Euclidean space is a space of n-dimensional vectors of real numbers. Manifold is a space (set of) vectors that for any point  of it small area around it could be transformed (smoothly and reversibly) to ball of Euclidean space. So manifold is locally almost Euclidean, but globally generally is not. That mean that every small area of manifold can be approximated by hyperplane (the smaller area, the more precise), but globally it can not be. Manifolds are introduced in differential topology and differential geometry.  
  > **username_0**: @mirror2image You should have described what an Euclidean space is. What are the necessary properties that a space must have to be considered Euclidean? You should also explain what you mean by "locally" and "globally".  
  > **username_3**: @nbro I purposefully didn't go into detail and tried explain intuition behind it. Formal definitions would be just copy-past from textbooks. Euclidean space is just  R^n which used in analysis and linear algebra. Locally and globally in depth - that require  some differential topology, where each explanation require more explanations until all the textbook is copied....  
  > **username_0**: I think your answer might be unnecessarily long and complex. For a long overview, I will just look at the paper I mentioned. I think you should try to compress your answer and just leave the key points/insights regarding GDL on manifolds. Note that this does not mean that you can't try to give an explanation of manifolds!  
  > **username_4**: @Bluepon: I'd be very very interested in working in geometric machine learning and currently looking for a postdoc in France. I've a PhD in differential geometry and also previous postdoctoral experience working in machine learning. Any idea if some professor does it in France? Thanks a lot!  
  
  ---
  
  ### Answer by username_5 on 2021-08-25. Score: 0
  
  I didn't read the paper in depth, but one example of where assumptions of Euclidean space are made in the design of the networks are with ConvNets in image processing.
  
  Specifically, Euclidean spaces are transformationally invariant, meaning that $d(a,b) = d(a+c,b+c)$. Each convolution layer iterates over the image with a certain amount of stride, which guarantees a certain amount of translational invariance in exchange for a smaller size of network parameters.
  
  ConvNets would not naively work for a dataset which comes from, say, a graph, because its not clear how to stride over it, as a possible example.
  
  > **Comments:**
  > **username_0**: Maybe this would be the answer to [my other related question](https://ai.stackexchange.com/q/11226/2444)? But I've just noticed that you also provided an answer there.  
  
  ---
  
  