Answers.Body: <p><strong>tl;dr</strong> I always like to think of Neural Networks as a generalization of <strong>logistic regression</strong>. </p>
  
  <p>I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks.</p>
  
  <p>A perceptron is essentially a simple binary logistic regressor (if you threshold the output). If you have many perceptrons that share the same input (i.e. a layer in a neural network), you can think of it as a multi-class logistic regressor. Now, by stacking one such layer after an other, you create a Multi-Layer Perceptron (MLP), which is a Neural Network with two layers. There is equivalent to two multi-class logistic regressors stacked one after the other. One notable thing that changes is the training technique here, i.e. backpropagation (because you don't have direct access to the targets from the hidden layer). Another thing that can change is the activation function (it's not always sigmoid in Neural Networks)</p>
  
  <p>Introduce sparse connectivity and weight sharing and you get a Convolutional Neural Network. Add a connection from a layer to its self (for the next timestep) and you get a Recurrent Neural Network. Likewise, you can reproduce any Neural Network through this reasoning.</p>
  
  <p>I know this is an over-simplified way of presenting them, but I think you get the point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-20T01:53:34.830Z
Answers.LastActivityDate: 2019-10-20T01:53:34.830Z
Answers.UserDisplayName: Djib2011
Answers.Body: <p>One way to view a neural network is as a series of linear transformations. </p>
  
  <p>You take a bunch of data points and look at it from a different perspective from a different space. You apply some non linear function on the data points like, ReLU, sigmoid etc. Now you repeat the same process of looking from a different space.</p>
  
  <p>Our goal is to look at it from a point where things starts looking right for our tasks. These linear transformations is what the network has to optimise. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-30T07:53:23.563Z
Answers.Text: That is to say, one neuron in a layer makes one coordinate, and the resulting points are then sharply differentiated by a non-linear function. So, approximation will be almost linear at some areas that are not covered by the network's "memory", and quite non-linear at other areas where the "lines of separation" are drawn and fixed in the network's weights. As the points on the second layer correspond to whole areas on the first layer, the resulting "lines of separation" may take on all kinds of shapes, not just a direct line as on the first layer..
Answers.UserDisplayName: Evgeniy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-29T04:59:12.923Z
Answers.LastActivityDate: 2019-10-29T04:59:12.923Z
Answers.UserDisplayName: Sooryakiran Pallikulathil
Answers.Body: <p>A good way of looking at it would be understanding neural networks mathematically, i.e. purely on the basis of the fact that you're just trying to fit a function and solve an optimisation problem (<em>apart from looking at it as multiple units of logistic regression</em>). </p>
  
  <p>Say we want to approximate a function <span class="math-container">$y =f_w(x)$</span> with <span class="math-container">$x \in D$</span>, where <span class="math-container">$D$</span> is our domain-space. We want this function to map to <span class="math-container">$C$</span>, our co-domain, with all the values the function ends up taking being the set <span class="math-container">$y \in R$</span>, our range. Essentially we frame <span class="math-container">$f(x)$</span> as a sequence of operations (<em>What operation should be done where is got from common-practice, intuition, and insight mostly gained from experience</em>) assuming that when the right parameters are used for these operations we will arrive at a very reasonable approximation of the function.</p>
  
  <p>We initialise the parameters with whatever values we want initially (<em>usually random</em>), calling this parameter-space <span class="math-container">$W$</span>. The essential idea would be frame another function <span class="math-container">$L(f_w(x), \hat{y})$</span> called the loss function which we want to minimise. This acts as a test to how good our function is - since our function parameters were initially random, the error between the function approximations and the actual range values for known points (training set) are estimated. These estimated error values and its gradient is then used by back-propagation where <span class="math-container">$w_{init}\in W$</span> is updated to another <span class="math-container">$w_{1}\in W$</span>, where <span class="math-container">$w_1$</span> is calculated by moving on <span class="math-container">$L$</span> in the direction of decreasing gradient, in hopes of reaching the loss functions minima.  </p>
  
  <p>Simplifying, essentially all you want to do is find a <span class="math-container">$y=f_w(x)$</span> where parameters <span class="math-container">$w$</span> are to be chosen such that <span class="math-container">$L(f_w(x), \hat{y})$</span> is minimised for the training set.</p>
  
  <p>Even though this is a very rough idea of neural networks, such a direction in thinking can especially be useful when studying generative networks and other problems where the problem has to be formulated mathematically before being able to approach it. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-29T06:19:31.340Z
Answers.LastActivityDate: 2019-10-29T06:19:31.340Z
Answers.UserDisplayName: ashenoy
Body: <p>Is there a way to understand, for instance, a multi-layered perceptron without hand-waving about them being similar to brains, etc?</p>
  <p>For example, it is obvious that what a perceptron does is approximating a function; there might be many other ways, given a labelled dataset, to find the separation of the input area into smaller areas that correspond to the labels; however, these ways would probably be computationally rather ineffective, which is why they cannot be practically used. However, it seems that the iterative approach of finding such areas of separation may give a huge speed-up in many cases; then, natural questions arise why this speed-up may be possible, how it happens and in which cases.</p>
  <p>One could be sure that this question was investigated. If anyone could shed any light on the history of this question, I would be very grateful.</p>
  <p>So, why are neural networks useful and what do they do? I mean, from the practical and mathematical standpoint, without relying on the concept of &quot;brain&quot; or &quot;neurons&quot; which can explain nothing at all.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-19T18:51:29.430Z
Comments.Text: Yes, thank you, that is exactly what (part of) my question is about. In fact, I assume that I may not understand the answers, but I still think this question has its place here. I wondered whether the structure of NNs could be motivated differently, by different ideas, I mean if we forget all about brains and just have a problem at hand, either to approximate a function or to label data clusters.
Comments.UserDisplayName: Evgeniy
Tags: <neural-networks><machine-learning><function-approximation><history>
Title: Is there a way to understand neural networks without using the concept of brain?
UserDisplayName: Evgeniy
ThreadText: # Is there a way to understand neural networks without using the concept of brain?
  
  (Asked by: username_0 on 2019-10-19)
  
  Is there a way to understand, for instance, a multi-layered perceptron without hand-waving about them being similar to brains, etc?
  
  For example, it is obvious that what a perceptron does is approximating a function; there might be many other ways, given a labelled dataset, to find the separation of the input area into smaller areas that correspond to the labels; however, these ways would probably be computationally rather ineffective, which is why they cannot be practically used. However, it seems that the iterative approach of finding such areas of separation may give a huge speed-up in many cases; then, natural questions arise why this speed-up may be possible, how it happens and in which cases.
  
  One could be sure that this question was investigated. If anyone could shed any light on the history of this question, I would be very grateful.
  
  So, why are neural networks useful and what do they do? I mean, from the practical and mathematical standpoint, without relying on the concept of "brain" or "neurons" which can explain nothing at all.
  
  **Tags:** neural-networks, machine-learning, function-approximation, history
  > **Comments:**
  > **username_0**: Yes, thank you, that is exactly what (part of) my question is about. In fact, I assume that I may not understand the answers, but I still think this question has its place here. I wondered whether the structure of NNs could be motivated differently, by different ideas, I mean if we forget all about brains and just have a problem at hand, either to approximate a function or to label data clusters.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-10-20. Score: 12
  
  **tl;dr** I always like to think of Neural Networks as a generalization of **logistic regression**.
  
  I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks.
  
  A perceptron is essentially a simple binary logistic regressor (if you threshold the output). If you have many perceptrons that share the same input (i.e. a layer in a neural network), you can think of it as a multi-class logistic regressor. Now, by stacking one such layer after an other, you create a Multi-Layer Perceptron (MLP), which is a Neural Network with two layers. There is equivalent to two multi-class logistic regressors stacked one after the other. One notable thing that changes is the training technique here, i.e. backpropagation (because you don't have direct access to the targets from the hidden layer). Another thing that can change is the activation function (it's not always sigmoid in Neural Networks)
  
  Introduce sparse connectivity and weight sharing and you get a Convolutional Neural Network. Add a connection from a layer to its self (for the next timestep) and you get a Recurrent Neural Network. Likewise, you can reproduce any Neural Network through this reasoning.
  
  I know this is an over-simplified way of presenting them, but I think you get the point.
  
  
  ---
  
  ### Answer by username_2 on 2019-10-29. Score: 3
  
  One way to view a neural network is as a series of linear transformations.
  
  You take a bunch of data points and look at it from a different perspective from a different space. You apply some non linear function on the data points like, ReLU, sigmoid etc. Now you repeat the same process of looking from a different space.
  
  Our goal is to look at it from a point where things starts looking right for our tasks. These linear transformations is what the network has to optimise.
  
  > **Comments:**
  > **username_0**: That is to say, one neuron in a layer makes one coordinate, and the resulting points are then sharply differentiated by a non-linear function. So, approximation will be almost linear at some areas that are not covered by the network's "memory", and quite non-linear at other areas where the "lines of separation" are drawn and fixed in the network's weights. As the points on the second layer correspond to whole areas on the first layer, the resulting "lines of separation" may take on all kinds of shapes, not just a direct line as on the first layer..  
  
  ---
  
  ### Answer by username_3 on 2019-10-29. Score: 0
  
  A good way of looking at it would be understanding neural networks mathematically, i.e. purely on the basis of the fact that you're just trying to fit a function and solve an optimisation problem (*apart from looking at it as multiple units of logistic regression*).
  
  Say we want to approximate a function $y =f\_w(x)$ with $x \in D$, where $D$ is our domain-space. We want this function to map to $C$, our co-domain, with all the values the function ends up taking being the set $y \in R$, our range. Essentially we frame $f(x)$ as a sequence of operations (*What operation should be done where is got from common-practice, intuition, and insight mostly gained from experience*) assuming that when the right parameters are used for these operations we will arrive at a very reasonable approximation of the function.
  
  We initialise the parameters with whatever values we want initially (*usually random*), calling this parameter-space $W$. The essential idea would be frame another function $L(f\_w(x), \hat{y})$ called the loss function which we want to minimise. This acts as a test to how good our function is - since our function parameters were initially random, the error between the function approximations and the actual range values for known points (training set) are estimated. These estimated error values and its gradient is then used by back-propagation where $w\_{init}\in W$ is updated to another $w\_{1}\in W$, where $w\_1$ is calculated by moving on $L$ in the direction of decreasing gradient, in hopes of reaching the loss functions minima.
  
  Simplifying, essentially all you want to do is find a $y=f\_w(x)$ where parameters $w$ are to be chosen such that $L(f\_w(x), \hat{y})$ is minimised for the training set.
  
  Even though this is a very rough idea of neural networks, such a direction in thinking can especially be useful when studying generative networks and other problems where the problem has to be formulated mathematically before being able to approach it.
  
  
  ---
  
  