Answers.Body: <p>The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.</p>
  <p>We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.</p>
  <p>We can control the height of the tower with the weights.</p>
  <p>First in 2d:</p>
  <p><a href="https://i.sstatic.net/VRGzb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VRGzb.png" alt="enter image description here" /></a></p>
  <p>Unfortunately the closer we put this towers together the more they will start two influence each other.</p>
  <p><a href="https://i.sstatic.net/FyRwL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FyRwL.png" alt="enter image description here" /></a></p>
  <p>But we can counter that with a negative tower between them.
  <a href="https://i.sstatic.net/isxkZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/isxkZ.png" alt="enter image description here" /></a></p>
  <p>Now in 3d:</p>
  <p><a href="https://i.sstatic.net/KKUs3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KKUs3.png" alt="enter image description here" /></a></p>
  <p>This Answer is work in progress I will update it when i find out something new.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T12:20:27.273Z
Answers.LastActivityDate: 2021-01-21T12:25:40.363Z
Answers.LastEditDate: 2021-01-21T12:25:40.363Z
Answers.UserDisplayName: KoKlA
Answers.Body: <p>I think you misunderstood that part of the proof: you first need a limit on the number of neurons to get closer and closer to a cylinder. You are keeping them constant at 1000, thus indeed not getting any closer to the cylinder and exponential vanishing behavior.</p>
  <p>Once you have the &quot;epsilon-perfect&quot; circles/cylinders, then you make them smaller and smaller, thus needing more and more copies of the &quot;1-circle&quot; setup.</p>
  <p>My understanding is that this proof has those two numbers going to infinity: neurons-to-approximate-cylinder, and number-of-cylinders. You took into account the latter, but not the former.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-20T19:16:58.137Z
Answers.LastActivityDate: 2022-05-20T19:48:37.650Z
Answers.LastEditDate: 2022-05-20T19:48:37.650Z
Answers.UserDisplayName: etal
Body: <p>There are proofs for the universal approximation theorem with just 1 hidden layer.</p>
  <p>The proof goes like this:</p>
  <ol>
  <li><p>Create a &quot;bump&quot; function using 2 neurons.</p>
  </li>
  <li><p>Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.</p>
  </li>
  <li><p>Decrease the step/radius to a very small value in order to approximate a cylinder. <strong>This is what I'm not convinced of</strong></p>
  </li>
  <li><p>Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like <a href="https://upload.wikimedia.org/wikipedia/commons/thumb/e/ee/15_circles_in_a_square.svg/1024px-15_circles_in_a_square.svg.png" rel="nofollow noreferrer">this</a>.</p>
  </li>
  </ol>
  <p>In <a href="http://www.youtube.com/watch?v=lkha188L4Gs&amp;t=42m0s" rel="nofollow noreferrer">this video, minute 42</a>, the lecturer says</p>
  <blockquote>
  <p>In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.</p>
  </blockquote>
  <p>Here are the slides.</p>
  <p><a href="https://i.sstatic.net/HPvXo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HPvXo.png" alt="enter image description here" /></a></p>
  <p><a href="https://www.mathematik.uni-wuerzburg.de/fileadmin/10040900/2019/Seminar__Artificial_Neural_Network__24_9__.pdf" rel="nofollow noreferrer">Here is a pdf version</a> from another university, so you do not have to watch the video.</p>
  <p><strong>Why am I not convinced?</strong></p>
  <p>I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.</p>
  <p>Let's start with a simple tower of radius 0.1:</p>
  <p><a href="https://i.sstatic.net/mT75p.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mT75p.png" alt="enter image description here" /></a></p>
  <p>Now let's decrease the radius to 0.01:</p>
  <p><a href="https://i.sstatic.net/Rq73M.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Rq73M.png" alt="enter image description here" /></a></p>
  <p>Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.</p>
  <p>Let's zoom in:</p>
  <p><a href="https://i.sstatic.net/Oxjtq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Oxjtq.png" alt="enter image description here" /></a></p>
  <p>Let's decrease the radius to 0.0000001.</p>
  <p><a href="https://i.sstatic.net/T7YcQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/T7YcQ.png" alt="enter image description here" /></a></p>
  <p>Still not a perfect cylinder. In fact, the &quot;quality&quot; of the cylinder is the same.</p>
  <p>Python code to reproduce (requires NumPy and matplotlib): <a href="https://pastebin.com/CMXFXvNj" rel="nofollow noreferrer">https://pastebin.com/CMXFXvNj</a>.</p>
  <p>So my questions are:</p>
  <p><strong>Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?</strong></p>
  <p><strong>Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?</strong></p>
  <p>Both towers have the same shape</p>
  <p>Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .</p>
  <p>Vc = Volume Cylinder</p>
  <p>Vt = Volume Tower</p>
  <p>ratio(r) = Vc/Vt</p>
  <p>What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.</p>
  <p>So what they are saying is that: lim r -&gt; 0 for ratio(r) = 1
  But my experiments show that: lim r -&gt; 0 for ratio(r) = const and don't depend on the radius at all.</p>
  <p><strong>Q3 Preface</strong></p>
  <p>An objection i got multiple times once from <a href="https://ai.stackexchange.com/users/9947/duttaa">Dutta</a> and once from <a href="https://ai.stackexchange.com/users/1794/d-w">D.W</a> is that just decreasing the radious and plotting it isn't mathematical rigorous.</p>
  <p>So let's assume in the limit of r=0 it's really a perfect cylinder.</p>
  <p>One possible explanation for this would be that the limit is a special case and one can't approximate towards it</p>
  <p>But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.</p>
  <p><strong>Q3 So why should we even care about this then ?</strong></p>
  <p><strong>Further Clarifications</strong></p>
  <p>The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. <strong>I am NOT questioning the paper !</strong> But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T17:45:50.160Z
Comments.Text: What's your question?  I don't see a question here.  A question usually ends with a "?".  You can't verify or disprove a statement about what happens "in the limit" by looking at finite instances.
Comments.UserDisplayName: D.W.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T18:39:25.437Z
Comments.Text: There is a question mark in the title. Well if I decrease the width to a very small value the shape of the tower should change and get closer to a cylinder imo. If there doesn't happen any transformation of the shape there is no use for the universal approximation theorem. It's only useful if the tower gets closer to a cylinder when the step width decreases. If it's only a perfect cylinder when the step width is exactly zero, but a completly different shape when it's 0.0000000000001 above zero there is no use for it.
Comments.UserDisplayName: KoKlA
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T19:01:39.493Z
Comments.Text: Let me try to put it into mathematical terms as good as i can. Let;s define ts as the shape of the tower. ts depends on the number of neurons used, and on the step width(w). Let's ignore the neurons for now. So we have ts(w) . So you are saying that ts(w=0) is a perfect cylinder ? But ts(w=1) = ts(w=0.1) = ts(w=0.01) = ts(w=0.001) are all the same shape. So ts would be a discontinous function. Well, if this is really true, then what's the point of the Universal Approximation for a single hidden layer ?
Comments.UserDisplayName: KoKlA
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T19:57:38.297Z
Comments.Text: Could you [edit] the question to include a self-contained statement of the theorem, and then tell us what kind of answer you are looking for?  I don't really want to watch an external video to understand what you are asking; and we discourage questions that rely on us to click external links to understand what is being asked.  Slides and informal statements like "nearly perfect cylinder" are not a substitute for an actual mathematical exposition with a precise statement of the theorem.
Comments.UserDisplayName: D.W.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T20:00:09.897Z
Comments.Text: It seems like the fact that you have to zoom in farther and farther to see the inaccuracy in the approximation is an indication that there is a sense in which it is becoming a closer approximation to a perfect cylinder.  No, it will never be a perfect cylinder for any finite value, but that is not a contradiction.  I am not sure what kind of answer you are looking for, though, or exactly what the question is.
Comments.UserDisplayName: D.W.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T20:10:06.660Z
Comments.Text: When I zoom I can see that the shape of the tower is exactly the same. So it doesn't get any closer to a cylinder.  It's the same shape just smaller. There is no difference in the quality of the shape.
Comments.UserDisplayName: KoKlA
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T20:23:02.783Z
Comments.Text: So it's not becoming a closer approximation of a cylinder. I'm aware that it will never become a perfect cylinder BUT i expect the shape to change. BUT it does NOT change it's exactly the same shape just smaller.
Comments.UserDisplayName: KoKlA
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T20:24:00.700Z
Comments.Text: I'll update the question later to include some clarifications.
Comments.UserDisplayName: KoKlA
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-19T21:25:59.807Z
Comments.Text: I suspect you've misunderstood the claim.  I'll look forward to the revision.
Comments.UserDisplayName: D.W.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-21T14:35:37.663Z
Comments.Text: It's really unfortunate that you started a bounty. I didn't yet follow the discussions above, but it looks like you now have many different questions in this post. It's not a good thing to have multiple questions in the same post. You should focus on the simplest problem that you have that you need to solve before moving to the next and ask a question about that one. Everything else should be moved to the next post. Please, read [this](https://meta.stackexchange.com/a/39224/287113).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-21T14:37:41.527Z
Comments.Text: So, what I suggest that you do, although you already started a bounty, is to simplify as much as possible this post and ask the first simplest question that you need an answer to, while providing all the necessary context for us to be able to answer such a question. If you have further questions, you should ask them in separate posts. In any case, I understand that when someone is confused about something, often, we don't really know what the best first question to ask is, and that's why often we end up asking multiple questions (this also happened to me).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T12:34:14.620Z
Comments.Text: Hey @KoKia! Did this question ever get resolved? I realise that I have the exact same question. It seems to be me that the method of stacking lots of bumps doesn't yield a "circular tower", not matter how small you make the radius. Have a look at my question, which is the same as yours: https://datascience.stackexchange.com/questions/106905/constructing-circular-towers-to-show-that-single-hidden-layer-feedforward-neural
Comments.UserDisplayName: Just_a_fool
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T20:05:15.350Z
Comments.Text: @Just_a_fool Hey I did a lot of work related to this topic. I haven't had yet time to fully write up everything. But i am planning to do that in the future. I am going to have a look at your question soon.
Comments.UserDisplayName: KoKlA
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T10:33:44.983Z
Comments.Text: @KoKlA Amazing. Thank you for your willingness to share. I am looking forward to see what you have found out whenever its done!
Comments.UserDisplayName: Just_a_fool
Tags: <neural-networks><feedforward-neural-networks><universal-approximation-theorems>
Title: Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
UserDisplayName: KoKlA
ThreadText: # Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
  
  (Asked by: username_0 on 2021-01-11)
  
  There are proofs for the universal approximation theorem with just 1 hidden layer.
  
  The proof goes like this:
  
  1. Create a "bump" function using 2 neurons.
  2. Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.
  3. Decrease the step/radius to a very small value in order to approximate a cylinder. **This is what I'm not convinced of**
  4. Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like [this](https://upload.wikimedia.org/wikipedia/commons/thumb/e/ee/15_circles_in_a_square.svg/1024px-15_circles_in_a_square.svg.png).
  
  In [this video, minute 42](http://www.youtube.com/watch?v=lkha188L4Gs&t=42m0s), the lecturer says
  
  > In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.
  
  Here are the slides.
  
  [![enter image description here](https://i.sstatic.net/HPvXo.png)](https://i.sstatic.net/HPvXo.png)
  
  [Here is a pdf version](https://www.mathematik.uni-wuerzburg.de/fileadmin/10040900/2019/Seminar__Artificial_Neural_Network__24_9__.pdf) from another university, so you do not have to watch the video.
  
  **Why am I not convinced?**
  
  I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.
  
  Let's start with a simple tower of radius 0.1:
  
  [![enter image description here](https://i.sstatic.net/mT75p.png)](https://i.sstatic.net/mT75p.png)
  
  Now let's decrease the radius to 0.01:
  
  [![enter image description here](https://i.sstatic.net/Rq73M.png)](https://i.sstatic.net/Rq73M.png)
  
  Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.
  
  Let's zoom in:
  
  [![enter image description here](https://i.sstatic.net/Oxjtq.png)](https://i.sstatic.net/Oxjtq.png)
  
  Let's decrease the radius to 0.0000001.
  
  [![enter image description here](https://i.sstatic.net/T7YcQ.png)](https://i.sstatic.net/T7YcQ.png)
  
  Still not a perfect cylinder. In fact, the "quality" of the cylinder is the same.
  
  Python code to reproduce (requires NumPy and matplotlib): <https://pastebin.com/CMXFXvNj>.
  
  So my questions are:
  
  **Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?**
  
  **Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?**
  
  Both towers have the same shape
  
  Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .
  
  Vc = Volume Cylinder
  
  Vt = Volume Tower
  
  ratio(r) = Vc/Vt
  
  What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.
  
  So what they are saying is that: lim r -> 0 for ratio(r) = 1
  But my experiments show that: lim r -> 0 for ratio(r) = const and don't depend on the radius at all.
  
  **Q3 Preface**
  
  An objection i got multiple times once from [Dutta](https://ai.stackexchange.com/users/9947/duttaa) and once from [D.W](https://ai.stackexchange.com/users/1794/d-w) is that just decreasing the radious and plotting it isn't mathematical rigorous.
  
  So let's assume in the limit of r=0 it's really a perfect cylinder.
  
  One possible explanation for this would be that the limit is a special case and one can't approximate towards it
  
  But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.
  
  **Q3 So why should we even care about this then ?**
  
  **Further Clarifications**
  
  The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. **I am NOT questioning the paper !** But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)
  
  **Tags:** neural-networks, feedforward-neural-networks, universal-approximation-theorems
  
  > **Comments:**
  > **username_1**: What's your question?  I don't see a question here.  A question usually ends with a "?".  You can't verify or disprove a statement about what happens "in the limit" by looking at finite instances.  
  > **username_0**: There is a question mark in the title. Well if I decrease the width to a very small value the shape of the tower should change and get closer to a cylinder imo. If there doesn't happen any transformation of the shape there is no use for the universal approximation theorem. It's only useful if the tower gets closer to a cylinder when the step width decreases. If it's only a perfect cylinder when the step width is exactly zero, but a completly different shape when it's 0.0000000000001 above zero there is no use for it.  
  > **username_0**: Let me try to put it into mathematical terms as good as i can. Let;s define ts as the shape of the tower. ts depends on the number of neurons used, and on the step width(w). Let's ignore the neurons for now. So we have ts(w) . So you are saying that ts(w=0) is a perfect cylinder ? But ts(w=1) = ts(w=0.1) = ts(w=0.01) = ts(w=0.001) are all the same shape. So ts would be a discontinous function. Well, if this is really true, then what's the point of the Universal Approximation for a single hidden layer ?  
  > **username_1**: Could you [edit] the question to include a self-contained statement of the theorem, and then tell us what kind of answer you are looking for?  I don't really want to watch an external video to understand what you are asking; and we discourage questions that rely on us to click external links to understand what is being asked.  Slides and informal statements like "nearly perfect cylinder" are not a substitute for an actual mathematical exposition with a precise statement of the theorem.  
  > **username_1**: It seems like the fact that you have to zoom in farther and farther to see the inaccuracy in the approximation is an indication that there is a sense in which it is becoming a closer approximation to a perfect cylinder.  No, it will never be a perfect cylinder for any finite value, but that is not a contradiction.  I am not sure what kind of answer you are looking for, though, or exactly what the question is.  
  > **username_0**: When I zoom I can see that the shape of the tower is exactly the same. So it doesn't get any closer to a cylinder.  It's the same shape just smaller. There is no difference in the quality of the shape.  
  > **username_0**: So it's not becoming a closer approximation of a cylinder. I'm aware that it will never become a perfect cylinder BUT i expect the shape to change. BUT it does NOT change it's exactly the same shape just smaller.  
  > **username_0**: I'll update the question later to include some clarifications.  
  > **username_1**: I suspect you've misunderstood the claim.  I'll look forward to the revision.  
  > **username_2**: It's really unfortunate that you started a bounty. I didn't yet follow the discussions above, but it looks like you now have many different questions in this post. It's not a good thing to have multiple questions in the same post. You should focus on the simplest problem that you have that you need to solve before moving to the next and ask a question about that one. Everything else should be moved to the next post. Please, read [this](https://meta.stackexchange.com/a/39224/287113).  
  > **username_2**: So, what I suggest that you do, although you already started a bounty, is to simplify as much as possible this post and ask the first simplest question that you need an answer to, while providing all the necessary context for us to be able to answer such a question. If you have further questions, you should ask them in separate posts. In any case, I understand that when someone is confused about something, often, we don't really know what the best first question to ask is, and that's why often we end up asking multiple questions (this also happened to me).  
  > **username_3**: Hey @KoKia! Did this question ever get resolved? I realise that I have the exact same question. It seems to be me that the method of stacking lots of bumps doesn't yield a "circular tower", not matter how small you make the radius. Have a look at my question, which is the same as yours: https://datascience.stackexchange.com/questions/106905/constructing-circular-towers-to-show-that-single-hidden-layer-feedforward-neural  
  > **username_0**: @username_3 Hey I did a lot of work related to this topic. I haven't had yet time to fully write up everything. But i am planning to do that in the future. I am going to have a look at your question soon.  
  > **username_3**: @username_0 Amazing. Thank you for your willingness to share. I am looking forward to see what you have found out whenever its done!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2021-01-21. Score: 1
  
  The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.
  
  We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.
  
  We can control the height of the tower with the weights.
  
  First in 2d:
  
  [![enter image description here](https://i.sstatic.net/VRGzb.png)](https://i.sstatic.net/VRGzb.png)
  
  Unfortunately the closer we put this towers together the more they will start two influence each other.
  
  [![enter image description here](https://i.sstatic.net/FyRwL.png)](https://i.sstatic.net/FyRwL.png)
  
  But we can counter that with a negative tower between them.
  [![enter image description here](https://i.sstatic.net/isxkZ.png)](https://i.sstatic.net/isxkZ.png)
  
  Now in 3d:
  
  [![enter image description here](https://i.sstatic.net/KKUs3.png)](https://i.sstatic.net/KKUs3.png)
  
  This Answer is work in progress I will update it when i find out something new.
  
  
  ---
  
  ### Answer by username_4 on 2022-05-20. Score: 0
  
  I think you misunderstood that part of the proof: you first need a limit on the number of neurons to get closer and closer to a cylinder. You are keeping them constant at 1000, thus indeed not getting any closer to the cylinder and exponential vanishing behavior.
  
  Once you have the "epsilon-perfect" circles/cylinders, then you make them smaller and smaller, thus needing more and more copies of the "1-circle" setup.
  
  My understanding is that this proof has those two numbers going to infinity: neurons-to-approximate-cylinder, and number-of-cylinders. You took into account the latter, but not the former.
  
  
  ---
  
  