Answers.Body: <p>First, what makes the neural network different than linear regression is the non-linearity (activation function), not the number of layers. So, a neural network with <span class="math-container">$n$</span> layers with no non-linearities is still the same as linear regression. Second, SVM finds the hyperplane of maximum margin. You are not guaranteed to find that hyperplane that has maximum margin with a neural network (using non-linearities).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T21:23:54.290Z
Answers.Text: okay fair enough, does this (see link below) then have any valid points? or will it ultimately just be an aproximation to the functionality of an SVM? https://stackoverflow.com/questions/54414392/convert-sklearn-svm-svc-classifier-to-keras-implementation
Answers.UserDisplayName: FeedMeInformation
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T21:30:25.613Z
Answers.Text: it will be an approximation. Unless you mathematically guarantee that is the maximum margin, it is just an approximation (That's why SVM is interesting, because it guarantees maximum margin, otherwise you stay with linear regression). Whenever you use a neural network, you can't guarantee you satisfy a constraint on the hyperplane found by the neural net. The math to do that doesn't exist yet.
Answers.UserDisplayName: Schach21
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T21:35:34.577Z
Answers.Text: okay that makes sense, cheers for the insight.
Answers.UserDisplayName: FeedMeInformation
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T21:02:34.380Z
Answers.LastActivityDate: 2021-03-10T15:56:25.953Z
Answers.LastEditDate: 2021-03-10T15:56:25.953Z
Answers.UserDisplayName: Schach21
Body: <p>A neural network without a hidden layer is the same as just linear regression.  </p>
  
  <p>If I then use squared hinge loss and encoporate the l2 regularisation term, is it fair to then call this network the same as a linear SVM?  </p>
  
  <p>Going by this assumption, then if I need to implement a multiclass SVM, i can just have n output nodes (where n is the number of classes). Would this then be equivalent to having n number of SVMs, similar to a one-vs-rest method?</p>
  
  <p>If I then wanted to encoporate a kernel into my SVM, could I then use an activation function or layer prior to the final output nodes (where I compute loss and add regularisation) which would then transfer this data into another feature plane the same as that of an SVM kernel? </p>
  
  <p>This is my current hunch, but would like some confirmation or correction where my understanding is incorrect. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-27T20:26:45.450Z
Comments.Text: "A neural network without a hidden layer is the same as just linear regression.", this is not true. This is only true if the activation functions are identities. So you'd better review your post and question.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><support-vector-machine>
Title: Is an SVM the same as a neural network without a hidden layer?
UserDisplayName: FeedMeInformation
ThreadText: # Is an SVM the same as a neural network without a hidden layer?
  
  (Asked by: username_0 on 2020-04-27)
  
  A neural network without a hidden layer is the same as just linear regression.
  
  If I then use squared hinge loss and encoporate the l2 regularisation term, is it fair to then call this network the same as a linear SVM?
  
  Going by this assumption, then if I need to implement a multiclass SVM, i can just have n output nodes (where n is the number of classes). Would this then be equivalent to having n number of SVMs, similar to a one-vs-rest method?
  
  If I then wanted to encoporate a kernel into my SVM, could I then use an activation function or layer prior to the final output nodes (where I compute loss and add regularisation) which would then transfer this data into another feature plane the same as that of an SVM kernel?
  
  This is my current hunch, but would like some confirmation or correction where my understanding is incorrect.
  
  **Tags:** neural-networks, machine-learning, support-vector-machine
  
  > **Comments:**
  > **username_1**: "A neural network without a hidden layer is the same as just linear regression.", this is not true. This is only true if the activation functions are identities. So you'd better review your post and question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-27. Score: 2 | Accepted answer
  
  First, what makes the neural network different than linear regression is the non-linearity (activation function), not the number of layers. So, a neural network with $n$ layers with no non-linearities is still the same as linear regression. Second, SVM finds the hyperplane of maximum margin. You are not guaranteed to find that hyperplane that has maximum margin with a neural network (using non-linearities).
  
  > **Comments:**
  > **username_0**: okay fair enough, does this (see link below) then have any valid points? or will it ultimately just be an aproximation to the functionality of an SVM? https://stackoverflow.com/questions/54414392/convert-sklearn-svm-svc-classifier-to-keras-implementation  
  > **username_2**: it will be an approximation. Unless you mathematically guarantee that is the maximum margin, it is just an approximation (That's why SVM is interesting, because it guarantees maximum margin, otherwise you stay with linear regression). Whenever you use a neural network, you can't guarantee you satisfy a constraint on the hyperplane found by the neural net. The math to do that doesn't exist yet.  
  > **username_0**: okay that makes sense, cheers for the insight.  
  
  ---
  
  