Answers.Body: <p><a href="https://arxiv.org/pdf/2004.06093.pdf" rel="nofollow noreferrer">Here</a> is a paper that explains why ReLU rules.</p>
  <p>What we want is to disentangle data of different classes. In order to do that, we need a discontinuous mapping for the data. ReLU easily allows for that. It is even better than LeakyReLU, sigmoid and tanh in that regard. Also, the reason any of the activations work is because of the floating point error, there is inadvertently a discontinuous mapping for the whole data. I have also explained it <a href="https://towardsdatascience.com/relu-rules-lets-understand-why-its-popularity-remains-unshaken-ccfe952fc5b1" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T14:51:25.153Z
Answers.Text: I think you meant to say non-homeomorphic mapping rather than discontinuous, as noted sect. 6.2 of the paper by Naitzat et al. you linked.  The $\mathrm{ReLU}$ mapping is continuous, of course.
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T17:20:48.500Z
Answers.Text: Is it? What happens at 0? If I can't map A to B, then, that creates a non-homeomorphic mapping, is that also not discontinuous? Isn't non-homeomorphism discontinuity in terms of topological spaces.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T18:08:06.783Z
Answers.Text: $\mathrm{ReLU}$ isn't differentiable at zero, but it is continuous. Roughly, a continuous function is one that "you can draw without lifting your pen" (the actual definition is more formal of course!). The reason that $\mathrm{ReLU}$ is not a homeomorphism is because it doesn't have a well-defined inverse (lots of values map to 0), and a homeomorphism is required to be invertible, and have a continuous inverse. That's why the authors contrast with the sigmoid, which does have an inverse.
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T18:19:25.607Z
Answers.Text: But, what it means physically is that sigmoid will always remain a lake and you would not be able to cut water. On the other hand, ReLU will be able to create those boundaries since it nullifies everything. Since, topology is the study of shapes. How would you explain this to a novice, the difference between homeomorphism and non-homeomorphism?
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T05:59:17.563Z
Answers.LastActivityDate: 2021-03-25T05:59:17.563Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I read Yann LeCun's paper <a href="https://www.researchgate.net/publication/2811922_Efficient_BackProp" rel="nofollow noreferrer">Efficient BackProp</a>, which was published in 2000. I looked for similar but more recent papers on <a href="https://arxiv.org/" rel="nofollow noreferrer">Arxiv</a>, but I have not yet found any.</p>
  <p><em>Are there relatively new research papers that describe how to make back-propagation more efficient?</em></p>
  <p>So, I am looking for papers similar to <a href="https://www.researchgate.net/publication/2811922_Efficient_BackProp" rel="nofollow noreferrer">Efficient Backprop</a> by LeCun but newer. The papers could describe why ReLU now &quot;dominates&quot; tanh or even sigmoid (but tanh was Yann's favorite, as explained in the paper). ReLU is just one thing I am interested in, but the paper could also analyze e.g. the inputs from a statistical standpoint.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-25T00:33:21.990Z
Comments.Text: A related question https://stats.stackexchange.com/questions/126238/what-are-the-advantages-of-relu-over-sigmoid-function-in-deep-neural-networks
Comments.UserDisplayName: user76284
Tags: <reference-request><backpropagation>
Title: Are there relatively new research papers that describe how to make back-propagation more efficient?
UserDisplayName: Good Luck
ThreadText: # Are there relatively new research papers that describe how to make back-propagation more efficient?
  
  (Asked by: username_0 on 2021-03-24)
  
  I read Yann LeCun's paper [Efficient BackProp](https://www.researchgate.net/publication/2811922_Efficient_BackProp), which was published in 2000. I looked for similar but more recent papers on [Arxiv](https://arxiv.org/), but I have not yet found any.
  
  *Are there relatively new research papers that describe how to make back-propagation more efficient?*
  
  So, I am looking for papers similar to [Efficient Backprop](https://www.researchgate.net/publication/2811922_Efficient_BackProp) by LeCun but newer. The papers could describe why ReLU now "dominates" tanh or even sigmoid (but tanh was Yann's favorite, as explained in the paper). ReLU is just one thing I am interested in, but the paper could also analyze e.g. the inputs from a statistical standpoint.
  
  **Tags:** reference-request, backpropagation
  > **Comments:**
  > **username_1**: A related question https://stats.stackexchange.com/questions/126238/what-are-the-advantages-of-relu-over-sigmoid-function-in-deep-neural-networks  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-25. Score: 1
  
  [Here](https://arxiv.org/pdf/2004.06093.pdf) is a paper that explains why ReLU rules.
  
  What we want is to disentangle data of different classes. In order to do that, we need a discontinuous mapping for the data. ReLU easily allows for that. It is even better than LeakyReLU, sigmoid and tanh in that regard. Also, the reason any of the activations work is because of the floating point error, there is inadvertently a discontinuous mapping for the whole data. I have also explained it [here](https://towardsdatascience.com/relu-rules-lets-understand-why-its-popularity-remains-unshaken-ccfe952fc5b1).
  
  > **Comments:**
  > **username_3**: I think you meant to say non-homeomorphic mapping rather than discontinuous, as noted sect. 6.2 of the paper by Naitzat et al. you linked.  The $\mathrm{ReLU}$ mapping is continuous, of course.  
  > **username_2**: Is it? What happens at 0? If I can't map A to B, then, that creates a non-homeomorphic mapping, is that also not discontinuous? Isn't non-homeomorphism discontinuity in terms of topological spaces.  
  > **username_3**: $\mathrm{ReLU}$ isn't differentiable at zero, but it is continuous. Roughly, a continuous function is one that "you can draw without lifting your pen" (the actual definition is more formal of course!). The reason that $\mathrm{ReLU}$ is not a homeomorphism is because it doesn't have a well-defined inverse (lots of values map to 0), and a homeomorphism is required to be invertible, and have a continuous inverse. That's why the authors contrast with the sigmoid, which does have an inverse.  
  > **username_2**: But, what it means physically is that sigmoid will always remain a lake and you would not be able to cut water. On the other hand, ReLU will be able to create those boundaries since it nullifies everything. Since, topology is the study of shapes. How would you explain this to a novice, the difference between homeomorphism and non-homeomorphism?  
  
  ---
  
  