Answers.Body: <p>Firstly, I invite you to take this answer with a grain of salt (and possibly suggest edits) as I am not specifically familiar with the issue of singularities myself. Also, I appologise for all the links being at the bottom of the answer, I am unable to reuse links throughout my answer due to my reputation on this particular exchange.</p>
  
  <hr>
  
  <p>This appears to refer to singular points in Algebraic Geometry. At singular points, <strong>no singlar tagent</strong> can be defined using the usual methods. This is as opposed to regular (i.e non-singular points), where one (and only one) tangent can be regularly defined.</p>
  
  <h2>The Easy Answer</h2>
  
  <p>In simpler terms, and particularly with respect to neural networks, it refers to the points on a hypersurface (e.g. a line, plane, surface, etc) at which the gradient/slope is ambigious.</p>
  
  <p>I've taken example of this from the wikipedia page on singular points.
  <a href="https://i.sstatic.net/JgjR3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JgjR3.png" alt="enter image description here"></a></p>
  
  <p>In the above image, the hyperspace is a line. On this line, there is a singularity at (0,0), where the line crosses itself. At this point, there are two slopes (one going up, and the other going down). There is no way to choose between the slopes without applying arbatrary rules, so the gradient is ambigous.</p>
  
  <p>These singularities are problematic for some of the algorithms used in deep learning, such as Gradient Descent, which uses the the derivative of objective functions (e.g. loss functions) to search for the weights/parameters that yield the lowest error.</p>
  
  <h2>Gradient Descent In This Example</h2>
  
  <p>For anyone unfamiliar with gradient descent, the problem with singularaties can be understood with the following example:</p>
  
  <p>Say that the above line represents a path. Starting at the top right, you want to find the lowest point on the path. At any time, all you can know is the slope at the current point. You can turn left or right freely, but will never move upwards, so you will stop as soon as you reach the bottom of a valley.</p>
  
  <p>Initially moving down and to the left, you will eventually reach the singularity and need to choose to go either left or right. Because you have no information about the path beyond your current location, you have no way of knowing which way leads to the bottom of the path.</p>
  
  <p>If you choose at random, you have a 50% chance of getting stuck at in the valley at the bottom of the loop on the left of the path.</p>
  
  <p>You could impose a rule like "always follow the gradient most different to the current gradient", then you'd go right, and find the true lowest point. However, if the path was slightly different (e.g. if it curved up at (0.25, 0.25) this would be bad, as the correct path would then be on the left.</p>
  
  <p>In practice, there is no way of knowing which way is better without a deep understanding of the hypersurface of your objective functions (and if you knew that much about the objective function, you likely wouldn't need to use machine learning in the first place).</p>
  
  <h2>Research/Resources</h2>
  
  <p>I discovered this in order to answer this questions so, as justification, here is the process and resources I used to come to this conclusion:</p>
  
  <p>I dug through the references in the paper you linked. Of particular relevance was <a href="https://pdfs.semanticscholar.org/f37a/392eac50939f73fb5e6f5445fcf0dacb9449.pdf" rel="nofollow noreferrer">Amari et al. (2006)</a>, which mentions <a href="https://en.wikipedia.org/wiki/Resolution_of_singularities#Hironaka&#39;s_method_2" rel="nofollow noreferrer">Hironaka’s
  theorem of singularity resolution</a>.</p>
  
  <p>In the introductory paragraph, the wikipedia page mentions the concept of Non-Singularity Variety, and links to <a href="https://en.wikipedia.org/wiki/Singular_point_of_an_algebraic_variety" rel="nofollow noreferrer">this page on singular points in Algebraic Geometry</a>.</p>
  
  <p>The example on singular points article is analogous to a 1d <a href="https://en.wikipedia.org/wiki/Resolution_of_singularities#Hironaka&#39;s_method_2" rel="nofollow noreferrer">hypersurface</a> version of many of the figures in Amari et al. (e.g. figures 1,2,4,5, and 6), and the discussion therein generally centers around the point at which the shapes converge.</p>
  
  <p>My answer was composed based on this discovery process, as well my prior-knowledge of <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="nofollow noreferrer">Gradient-Descent</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-19T06:39:36.100Z
Answers.LastActivityDate: 2018-05-19T07:04:45.357Z
Answers.LastEditDate: 2018-05-19T07:04:45.357Z
Answers.UserDisplayName: Cosmo
Body: <p>The paper <a href="https://arxiv.org/pdf/1701.09175v8.pdf" rel="nofollow noreferrer">Skip connections eliminate singularities</a> explains the use of skip connections to break the singularity in deep networks, but I have not fully understood what a <strong>singularity</strong> is.</p>
  <p>Any easy-understanding explanation?</p>
  
Tags: <neural-networks><deep-learning><terminology><deep-neural-networks>
Title: What's the definition of "singularity" in the context of neural networks?
UserDisplayName: Saeed
ThreadText: # What's the definition of "singularity" in the context of neural networks?
  
  (Asked by: username_0 on 2018-05-18)
  
  The paper [Skip connections eliminate singularities](https://arxiv.org/pdf/1701.09175v8.pdf) explains the use of skip connections to break the singularity in deep networks, but I have not fully understood what a **singularity** is.
  
  Any easy-understanding explanation?
  
  **Tags:** neural-networks, deep-learning, terminology, deep-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-05-19. Score: 3
  
  Firstly, I invite you to take this answer with a grain of salt (and possibly suggest edits) as I am not specifically familiar with the issue of singularities myself. Also, I appologise for all the links being at the bottom of the answer, I am unable to reuse links throughout my answer due to my reputation on this particular exchange.
  
  ---
  
  This appears to refer to singular points in Algebraic Geometry. At singular points, **no singlar tagent** can be defined using the usual methods. This is as opposed to regular (i.e non-singular points), where one (and only one) tangent can be regularly defined.
  
  ## The Easy Answer
  
  In simpler terms, and particularly with respect to neural networks, it refers to the points on a hypersurface (e.g. a line, plane, surface, etc) at which the gradient/slope is ambigious.
  
  I've taken example of this from the wikipedia page on singular points.
  [![enter image description here](https://i.sstatic.net/JgjR3.png)](https://i.sstatic.net/JgjR3.png)
  
  In the above image, the hyperspace is a line. On this line, there is a singularity at (0,0), where the line crosses itself. At this point, there are two slopes (one going up, and the other going down). There is no way to choose between the slopes without applying arbatrary rules, so the gradient is ambigous.
  
  These singularities are problematic for some of the algorithms used in deep learning, such as Gradient Descent, which uses the the derivative of objective functions (e.g. loss functions) to search for the weights/parameters that yield the lowest error.
  
  ## Gradient Descent In This Example
  
  For anyone unfamiliar with gradient descent, the problem with singularaties can be understood with the following example:
  
  Say that the above line represents a path. Starting at the top right, you want to find the lowest point on the path. At any time, all you can know is the slope at the current point. You can turn left or right freely, but will never move upwards, so you will stop as soon as you reach the bottom of a valley.
  
  Initially moving down and to the left, you will eventually reach the singularity and need to choose to go either left or right. Because you have no information about the path beyond your current location, you have no way of knowing which way leads to the bottom of the path.
  
  If you choose at random, you have a 50% chance of getting stuck at in the valley at the bottom of the loop on the left of the path.
  
  You could impose a rule like "always follow the gradient most different to the current gradient", then you'd go right, and find the true lowest point. However, if the path was slightly different (e.g. if it curved up at (0.25, 0.25) this would be bad, as the correct path would then be on the left.
  
  In practice, there is no way of knowing which way is better without a deep understanding of the hypersurface of your objective functions (and if you knew that much about the objective function, you likely wouldn't need to use machine learning in the first place).
  
  ## Research/Resources
  
  I discovered this in order to answer this questions so, as justification, here is the process and resources I used to come to this conclusion:
  
  I dug through the references in the paper you linked. Of particular relevance was [Amari et al. (2006)](https://pdfs.semanticscholar.org/f37a/392eac50939f73fb5e6f5445fcf0dacb9449.pdf), which mentions [Hironaka’s
  theorem of singularity resolution](https://en.wikipedia.org/wiki/Resolution_of_singularities#Hironaka's_method_2).
  
  In the introductory paragraph, the wikipedia page mentions the concept of Non-Singularity Variety, and links to [this page on singular points in Algebraic Geometry](https://en.wikipedia.org/wiki/Singular_point_of_an_algebraic_variety).
  
  The example on singular points article is analogous to a 1d [hypersurface](https://en.wikipedia.org/wiki/Resolution_of_singularities#Hironaka's_method_2) version of many of the figures in Amari et al. (e.g. figures 1,2,4,5, and 6), and the discussion therein generally centers around the point at which the shapes converge.
  
  My answer was composed based on this discovery process, as well my prior-knowledge of [Gradient-Descent](https://en.wikipedia.org/wiki/Gradient_descent)
  
  
  ---
  
  