Answers.Body: <p>I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different:</p>
  <ul>
  <li>The first image shows a linear classifier assigning scores for each of <span class="math-container">$C$</span> classes to an input image</li>
  <li>The second image shows a <strong>binary classifier</strong> (I think) scoring an item as ether white or purple (or something like that)</li>
  </ul>
  <p>Consider a <em>single</em> class in the linear classifier. The linear classifier can <em>only</em> draw a <em>single line</em> through <span class="math-container">$N$</span> dimensional space. Items on one side of this line have negative car scores. Items on the other side have positive car scores.</p>
  <p>For example:
  <a href="https://i.sstatic.net/4pN8V.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4pN8V.png" alt="enter image description here" /></a></p>
  <p>Here, the 2nd image in the row represents the SVM's attempt to extract all the features of a car into a single image. As images are &quot;closer&quot; to this image, the SVM will give them a higher car score. As images are farther away from this image, the SVM will give them a lower car score.</p>
  <p>The SVM cannot extract individual features, it has to smush them all into a single line in <span class="math-container">$D$</span> dimensional space (where <span class="math-container">$D$</span> is the number of pixels). That line can be represented as:</p>
  <p><a href="https://i.sstatic.net/ph3Ow.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ph3Ow.png" alt="enter image description here" /></a></p>
  <p>On the other hand. A multi-layer neural network, for example:
  <a href="https://i.sstatic.net/obiDn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/obiDn.png" alt="enter image description here" /></a></p>
  <p>Can now partition the space into a 100 different ways. The example is a bit convoluted,  since the neural net is classifying 1 thing into several buckets instead of doing binary classification, but the principle is the same. Maybe 10 out of a 100 neurons in the second layer are dedicated to partitioning the space of images with regard to cars.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T17:56:48.480Z
Answers.LastActivityDate: 2022-06-23T17:56:48.480Z
Answers.UserDisplayName: Foobar
Body: <p>My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity.</p>
  <p>However, here's where my understanding starts to break down. A classifier, like Softmax or SVM is considered to be a &quot;linear classifier&quot;. I'm using the definitions from <a href="https://cs231n.github.io/linear-classify/" rel="nofollow noreferrer">CS 231N</a>. There may be another definition of an SVM, but I'm not considering that.</p>
  <p>In a linear classifier we have the following: <span class="math-container">$W \cdot x$</span> where</p>
  <ul>
  <li><span class="math-container">$W$</span> is the weights</li>
  <li><span class="math-container">$x$</span> is the input vector.</li>
  </ul>
  <p>The output is a vector of scores. 1 score per label.</p>
  <p>An SVM classifier uses hinge loss to update the weights:</p>
  <p><a href="https://i.sstatic.net/MGrOf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MGrOf.png" alt="enter image description here" /></a></p>
  <p>A softmax classifier normalizes the output values using the softmax function and then uses cross-entropy loss to update the weights:</p>
  <p><a href="https://i.sstatic.net/llvft.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/llvft.png" alt="enter image description here" /></a></p>
  <p>From the lecture <a href="https://www.youtube.com/watch?v=8inugqHkfvE&amp;list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC&amp;index=3" rel="nofollow noreferrer">CS231n Winter 2016: Lecture 2: Data-driven approach, kNN, Linear Classification 1</a>, we have the following image to help visualize what a linear classifier does:</p>
  <p><a href="https://i.sstatic.net/Ybxbc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ybxbc.png" alt="Image about linear classifiers" /></a></p>
  <p>Essentially, each image can be considered a point in 3072 dimensional space, and we are drawing lines through this space. On one side of the &quot;car&quot; line, the car score for a given point will go up. On the other side of the &quot;car&quot; line, the car score for a given point will go down.</p>
  <p>However, this doesn't seem to be that much different from ReLU (Taken from this post: <a href="https://stats.stackexchange.com/questions/158549/why-are-activation-functions-needed-in-neural-networks">https://stats.stackexchange.com/questions/158549/why-are-activation-functions-needed-in-neural-networks</a>):</p>
  <p><img src="https://i.sstatic.net/hDrW2.png" alt="enter image description here" /></p>
  <p>So what is the fundamental difference between a &quot;linear&quot; classifier like Softmax / SVM and a multi-layer neural network? Why can't a SVM classifier learn any function but a neural network can?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-23T13:52:23.907Z
Comments.Text: In my view neural nets are more universal than SVMs: they can approximate any functions from one multi-dimensional space to another. They are often used as classifiers by adding SoftMax or some other trick. SVMs, on the other hand, are purely classifiers, but they can do non-linear classification using so called "kernel trick" (https://en.wikipedia.org/wiki/Support-vector_machine).
Comments.UserDisplayName: blamocur
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-23T13:58:32.483Z
Comments.Text: @blamocur Ideally, our site is not meant to share "views" or "opinions" but facts and evidence. Just because there's a lot of hype around NNs now, it doesn't mean that they are more universal than SVMs, whatever that means. See also [this](https://ai.stackexchange.com/q/12239/2444).
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><activation-functions><support-vector-machine><softmax>
Title: Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?
UserDisplayName: Foobar
ThreadText: # Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?
  
  (Asked by: username_0 on 2022-06-22)
  
  My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity.
  
  However, here's where my understanding starts to break down. A classifier, like Softmax or SVM is considered to be a "linear classifier". I'm using the definitions from [CS 231N](https://cs231n.github.io/linear-classify/). There may be another definition of an SVM, but I'm not considering that.
  
  In a linear classifier we have the following: $W \cdot x$ where
  
  * $W$ is the weights
  * $x$ is the input vector.
  
  The output is a vector of scores. 1 score per label.
  
  An SVM classifier uses hinge loss to update the weights:
  
  [![enter image description here](https://i.sstatic.net/MGrOf.png)](https://i.sstatic.net/MGrOf.png)
  
  A softmax classifier normalizes the output values using the softmax function and then uses cross-entropy loss to update the weights:
  
  [![enter image description here](https://i.sstatic.net/llvft.png)](https://i.sstatic.net/llvft.png)
  
  From the lecture [CS231n Winter 2016: Lecture 2: Data-driven approach, kNN, Linear Classification 1](https://www.youtube.com/watch?v=8inugqHkfvE&list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC&index=3), we have the following image to help visualize what a linear classifier does:
  
  [![Image about linear classifiers](https://i.sstatic.net/Ybxbc.png)](https://i.sstatic.net/Ybxbc.png)
  
  Essentially, each image can be considered a point in 3072 dimensional space, and we are drawing lines through this space. On one side of the "car" line, the car score for a given point will go up. On the other side of the "car" line, the car score for a given point will go down.
  
  However, this doesn't seem to be that much different from ReLU (Taken from this post: <https://stats.stackexchange.com/questions/158549/why-are-activation-functions-needed-in-neural-networks>):
  
  ![enter image description here](https://i.sstatic.net/hDrW2.png)
  
  So what is the fundamental difference between a "linear" classifier like Softmax / SVM and a multi-layer neural network? Why can't a SVM classifier learn any function but a neural network can?
  
  **Tags:** neural-networks, machine-learning, activation-functions, support-vector-machine, softmax
  
  > **Comments:**
  > **username_1**: In my view neural nets are more universal than SVMs: they can approximate any functions from one multi-dimensional space to another. They are often used as classifiers by adding SoftMax or some other trick. SVMs, on the other hand, are purely classifiers, but they can do non-linear classification using so called "kernel trick" (https://en.wikipedia.org/wiki/Support-vector_machine).  
  > **username_2**: @username_1 Ideally, our site is not meant to share "views" or "opinions" but facts and evidence. Just because there's a lot of hype around NNs now, it doesn't mean that they are more universal than SVMs, whatever that means. See also [this](https://ai.stackexchange.com/q/12239/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-23. Score: 1 | Accepted answer
  
  I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different:
  
  * The first image shows a linear classifier assigning scores for each of $C$ classes to an input image
  * The second image shows a **binary classifier** (I think) scoring an item as ether white or purple (or something like that)
  
  Consider a *single* class in the linear classifier. The linear classifier can *only* draw a *single line* through $N$ dimensional space. Items on one side of this line have negative car scores. Items on the other side have positive car scores.
  
  For example:
  [![enter image description here](https://i.sstatic.net/4pN8V.png)](https://i.sstatic.net/4pN8V.png)
  
  Here, the 2nd image in the row represents the SVM's attempt to extract all the features of a car into a single image. As images are "closer" to this image, the SVM will give them a higher car score. As images are farther away from this image, the SVM will give them a lower car score.
  
  The SVM cannot extract individual features, it has to smush them all into a single line in $D$ dimensional space (where $D$ is the number of pixels). That line can be represented as:
  
  [![enter image description here](https://i.sstatic.net/ph3Ow.png)](https://i.sstatic.net/ph3Ow.png)
  
  On the other hand. A multi-layer neural network, for example:
  [![enter image description here](https://i.sstatic.net/obiDn.png)](https://i.sstatic.net/obiDn.png)
  
  Can now partition the space into a 100 different ways. The example is a bit convoluted, since the neural net is classifying 1 thing into several buckets instead of doing binary classification, but the principle is the same. Maybe 10 out of a 100 neurons in the second layer are dedicated to partitioning the space of images with regard to cars.
  
  
  ---
  
  