Answers.Body: <p>The hinge loss/error function is the typical loss function used for <strong>binary classification</strong> (but it can also be extended to multi-class classification) in the context of <strong>support vector machines</strong>, although it can also be used in the context of neural networks, as described <a href="https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/" rel="nofollow noreferrer">here</a>.</p>
  <p>The hinge loss function is defined as follows</p>
  <p><span class="math-container">$$
  \ell(y) = \max(0, 1-t \cdot y) \tag{1}\label{1},
  $$</span>
  where</p>
  <ul>
  <li><span class="math-container">$t = \{-1, 1\}$</span> is the label (so, if your labels are in the set <span class="math-container">$\{0, 1 \}$</span>, you will have to first map them to <span class="math-container">$\{-1, 1\}$</span>)</li>
  <li><span class="math-container">$y$</span> is the output of the classifier (e.g. in the context of the linear SVM, <span class="math-container">$y=\mathbf{w} \cdot \mathbf{x}+b$</span>, where <span class="math-container">$\mathbf{w}$</span> and <span class="math-container">$b$</span> are the parameter of the hyper-plane)</li>
  </ul>
  <p>This means that the loss in equation \ref{1} is always non-negative. If you're familiar with the <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a>, this loss should look familiar to you. In fact, their plots are very similar.</p>
  <p>For more details, you probably should start with <a href="https://en.wikipedia.org/wiki/Hinge_loss" rel="nofollow noreferrer">the related Wikipedia article</a>, then maybe one of the many machine learning books that covers support vector machines, for example, <a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=345" rel="nofollow noreferrer">Pattern Recognition and Machine Learning</a> (2006) by  Christopher Bishop, chapter 7 (page 325).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-11T13:23:32.767Z
Answers.LastActivityDate: 2021-02-11T13:23:32.767Z
Answers.UserDisplayName: nbro
Body: <p>I came across the hinge loss function for training a neural network model, but I did not know the analytical form for the same.</p>
  <p>I can write the mean squared error loss function (which is more often used for regression) as</p>
  <p><span class="math-container">$$\sum\limits_{i=1}^{N}(y_i - \hat{y_i})^2$$</span></p>
  <p>where <span class="math-container">$y_i$</span> is the desired output in the dataset, <span class="math-container">$\hat{y_i}$</span> is the actual output by the model, and <span class="math-container">$N$</span> is the total number of instances in our dataset.</p>
  <p>Similarly, what is the (basic) expression for hinge loss function?</p>
  
Tags: <neural-networks><definitions><support-vector-machine><binary-classification><hinge-loss>
Title: What is the definition of the hinge loss function?
UserDisplayName: hanugm
ThreadText: # What is the definition of the hinge loss function?
  
  (Asked by: username_0 on 2021-02-11)
  
  I came across the hinge loss function for training a neural network model, but I did not know the analytical form for the same.
  
  I can write the mean squared error loss function (which is more often used for regression) as
  
  $$\sum\limits\_{i=1}^{N}(y\_i - \hat{y\_i})^2$$
  
  where $y\_i$ is the desired output in the dataset, $\hat{y\_i}$ is the actual output by the model, and $N$ is the total number of instances in our dataset.
  
  Similarly, what is the (basic) expression for hinge loss function?
  
  **Tags:** neural-networks, definitions, support-vector-machine, binary-classification, hinge-loss
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-11. Score: 3
  
  The hinge loss/error function is the typical loss function used for **binary classification** (but it can also be extended to multi-class classification) in the context of **support vector machines**, although it can also be used in the context of neural networks, as described [here](https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/).
  
  The hinge loss function is defined as follows
  
  $$
  \ell(y) = \max(0, 1-t \cdot y) \tag{1}\label{1},
  $$
  where
  
  * $t = \{-1, 1\}$ is the label (so, if your labels are in the set $\{0, 1 \}$, you will have to first map them to $\{-1, 1\}$)
  * $y$ is the output of the classifier (e.g. in the context of the linear SVM, $y=\mathbf{w} \cdot \mathbf{x}+b$, where $\mathbf{w}$ and $b$ are the parameter of the hyper-plane)
  
  This means that the loss in equation \ref{1} is always non-negative. If you're familiar with the [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), this loss should look familiar to you. In fact, their plots are very similar.
  
  For more details, you probably should start with [the related Wikipedia article](https://en.wikipedia.org/wiki/Hinge_loss), then maybe one of the many machine learning books that covers support vector machines, for example, [Pattern Recognition and Machine Learning](http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=345) (2006) by Christopher Bishop, chapter 7 (page 325).
  
  
  ---
  
  