Answers.Body: <p>In terms of the normal use cases for machine learning, the equation does not have much utility, because:</p>
  
  <blockquote>
    <p>Consider we have a curve <span class="math-container">$f(x)$</span> now if one wishes to  . . . </p>
  </blockquote>
  
  <p>In most AI problems, we don't usually have such a curve as input that can be treated analytically. For instance, there is no such input curve to describe a natural image received by a sensor. </p>
  
  <p>In the vast majority of cases in AI problems, the form of inputs - whether it is images, text, mapping data, robotic telemetry, is going to be multi-dimensional discrete samples from highly complex functions where we don't know the analytical form and can only construct approximations from a set of basis functions. The resulting combination of basis functions could be treated as continuous, integrated etc, but as it would have been constructed from discrete data, the end result would be a lot of computation to end up with something probably less accurate than working direct with the discrete samples. In a lot of cases, the raw data is discrete by definition (e.g. whether someone clicked on a link or replied to a message), so the form of <span class="math-container">$f(x)$</span> would be discrete by definition of the problem, and calculus not really applicable.</p>
  
  <p>There <em>might</em> be some interesting use cases in analog signal processing. Using your formula or a variation of it for instance it should be possible to create an analog neural network learning system - a robotic brain that worked with continuous signals and only contained analog components. Such a system would have some interesting advantages - speed of processing, probably low power consumption compared to digital approach, but the accuracy and precision might be lower. E.g. imagine a bot that could steer towards/away from light sources (compared to a digital one that might recognise the faces of people and steer towards/away from them). I would not be at all surprised to find that someone had done just that already, although I am not sure how to search for it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-04T16:40:00.653Z
Answers.LastActivityDate: 2019-01-04T16:45:50.137Z
Answers.LastEditDate: 2019-01-04T16:45:50.137Z
Answers.UserDisplayName: Neil Slater
Body: <h2>Background</h2>
  
  <p>My understanding is the input neurons seem to seem to compute a weighted sum moving from one layer to another. </p>
  
  <p><a href="https://i.sstatic.net/9tqAt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9tqAt.png" alt="enter image description here"></a>
  <span class="math-container">$$ \sum_i a_i w_i = a'_{k} $$</span></p>
  
  <p>But to compute this weighted sum the sum must be discrete. Is there any known method to compute the sum when the activation is a continuous function? Is the below formula of any consequence problems in artificial intelligence? Can anyone give a specific problem where it might be useful?</p>
  
  <h2>My Method</h2>
  
  <p>Let <span class="math-container">$b_r = \sum_{d \mid r} a_d\mu(\frac{m}{d})$</span>. We prove that if the <span class="math-container">$b_r$</span>'s are small enough, the result is true (where <span class="math-container">$\mu$</span> is the mobius function). </p>
  
  <blockquote>
    <p>Claim: If <span class="math-container">$\lim_{n \to \infty} \frac{\log^2(n)}{n}\sum_{r=1}^n |b_r| = 0$</span> and <span class="math-container">$f$</span> is smooth, then <span class="math-container">$$\lim_{k \to \infty} \lim_{n \to \infty} \sum_{r=1}^n a_rf\left(\frac{kr}{n}\right)\frac{k}{n} = \left(\lim_{s \to 1} \frac{1}{\zeta(s)}\sum_{r=1}^\infty \frac{a_r}{r^s}\right)\int_0^\infty f(x)dx.$$</span></p>
  </blockquote>
  
  <p>I will not go into the proof of this over but for those who are interested: <a href="https://math.stackexchange.com/questions/2888976/a-rough-proof-for-infinitesimals">https://math.stackexchange.com/questions/2888976/a-rough-proof-for-infinitesimals</a> I will merely state what the formula means:</p>
  
  <p>Consider we have a curve <span class="math-container">$f(x)$</span> now if one wishes to perform a weighted sum in the limiting case of this function. </p>
  
  <p><a href="https://i.sstatic.net/i0hCV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/i0hCV.png" alt="enter image description here"></a></p>
  
  <p>Consider the curve <span class="math-container">$f(x)$</span>. Then splitting it to <span class="math-container">$k/n= h$</span> intervals then adding the first strip (<span class="math-container">$d_1$</span> times): <span class="math-container">$  f(h) \cdot d_1$</span>. Then the second strip (<span class="math-container">$d_2$</span> times) <span class="math-container">$ f(2h) \cdot d_2$</span> times ... And so on . Hence. <span class="math-container">$d_r$</span> can be thought of as the weight at <span class="math-container">$f(rh)$</span>.</p>
  
Tags: <artificial-neuron><activation-functions>
Title: Method to compute the sum when the activation is a continuous function?
UserDisplayName: More Anonymous
ThreadText: # Method to compute the sum when the activation is a continuous function?
  
  (Asked by: username_0 on 2019-01-03)
  
  ## Background
  
  My understanding is the input neurons seem to seem to compute a weighted sum moving from one layer to another.
  
  [![enter image description here](https://i.sstatic.net/9tqAt.png)](https://i.sstatic.net/9tqAt.png)
  $$ \sum\_i a\_i w\_i = a'\_{k} $$
  
  But to compute this weighted sum the sum must be discrete. Is there any known method to compute the sum when the activation is a continuous function? Is the below formula of any consequence problems in artificial intelligence? Can anyone give a specific problem where it might be useful?
  
  ## My Method
  
  Let $b\_r = \sum\_{d \mid r} a\_d\mu(\frac{m}{d})$. We prove that if the $b\_r$'s are small enough, the result is true (where $\mu$ is the mobius function).
  
  > Claim: If $\lim\_{n \to \infty} \frac{\log^2(n)}{n}\sum\_{r=1}^n |b\_r| = 0$ and $f$ is smooth, then $$\lim\_{k \to \infty} \lim\_{n \to \infty} \sum\_{r=1}^n a\_rf\left(\frac{kr}{n}\right)\frac{k}{n} = \left(\lim\_{s \to 1} \frac{1}{\zeta(s)}\sum\_{r=1}^\infty \frac{a\_r}{r^s}\right)\int\_0^\infty f(x)dx.$$
  
  I will not go into the proof of this over but for those who are interested: <https://math.stackexchange.com/questions/2888976/a-rough-proof-for-infinitesimals> I will merely state what the formula means:
  
  Consider we have a curve $f(x)$ now if one wishes to perform a weighted sum in the limiting case of this function.
  
  [![enter image description here](https://i.sstatic.net/i0hCV.png)](https://i.sstatic.net/i0hCV.png)
  
  Consider the curve $f(x)$. Then splitting it to $k/n= h$ intervals then adding the first strip ($d\_1$ times): $ f(h) \cdot d\_1$. Then the second strip ($d\_2$ times) $ f(2h) \cdot d\_2$ times ... And so on . Hence. $d\_r$ can be thought of as the weight at $f(rh)$.
  
  **Tags:** artificial-neuron, activation-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-04. Score: 2
  
  In terms of the normal use cases for machine learning, the equation does not have much utility, because:
  
  > Consider we have a curve $f(x)$ now if one wishes to . . .
  
  In most AI problems, we don't usually have such a curve as input that can be treated analytically. For instance, there is no such input curve to describe a natural image received by a sensor.
  
  In the vast majority of cases in AI problems, the form of inputs - whether it is images, text, mapping data, robotic telemetry, is going to be multi-dimensional discrete samples from highly complex functions where we don't know the analytical form and can only construct approximations from a set of basis functions. The resulting combination of basis functions could be treated as continuous, integrated etc, but as it would have been constructed from discrete data, the end result would be a lot of computation to end up with something probably less accurate than working direct with the discrete samples. In a lot of cases, the raw data is discrete by definition (e.g. whether someone clicked on a link or replied to a message), so the form of $f(x)$ would be discrete by definition of the problem, and calculus not really applicable.
  
  There *might* be some interesting use cases in analog signal processing. Using your formula or a variation of it for instance it should be possible to create an analog neural network learning system - a robotic brain that worked with continuous signals and only contained analog components. Such a system would have some interesting advantages - speed of processing, probably low power consumption compared to digital approach, but the accuracy and precision might be lower. E.g. imagine a bot that could steer towards/away from light sources (compared to a digital one that might recognise the faces of people and steer towards/away from them). I would not be at all surprised to find that someone had done just that already, although I am not sure how to search for it.
  
  
  ---
  
  