Answers.Body: <p>There are different functions to pick from for your loss.
  The most common read online is the Mean Squared Error.</p>
  <p>It calculates how far the prediction is off from the actual value and squares it. All of them are summed up.</p>
  <p>So basically the Loss-Function &quot;generalizes&quot; the error over every prediction. The actual error in a case is just how far off it is.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-26T14:39:24.673Z
Answers.Text: In my code I have:
  `code`
  error=outputs-training_set_outputs
  #derivation calcs with error
  `code`
  So by your answer I understand that the MSE would be used as a metric of how well the model generalizes only?
Answers.UserDisplayName: user20170158
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-26T14:44:06.247Z
Answers.Text: The loss function is a way to evaluate your whole model, yes.
  However I don't get what your error should be. For me it looks like your error is just your output
Answers.UserDisplayName: SickerDude43
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-26T15:04:01.397Z
Answers.Text: if I use different loss functions in tensorflow, I do get different results, is this because the loss function is used in the optimizer? (rather than changing the error function error=y-y_hat)
Answers.UserDisplayName: user20170158
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T09:48:24.650Z
Answers.Text: The answer above is really good. However it's natural that different Loss Functions give you different results.
  
  You can calculate it by hand if you want, the Mean Squared Error is also often taught in school. It's totally doable. Just think of a small example, choose different functions and then calculate. You'll see easily why you'll get different results. Just keep in mind that having a smaller error is not always good. You generally want to avoid Over- and Underfitting. Also it depends on what you want to do. There is not just one solution in AI
Answers.UserDisplayName: SickerDude43
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-26T14:35:28.343Z
Answers.LastActivityDate: 2023-01-26T14:35:28.343Z
Answers.UserDisplayName: SickerDude43
Answers.Body: <p>A loss function is the function which can get the losses(differences) between a model's predictions and true values to evaluate how good a model is. *Loss function is also called Cost Function or Error Function.</p>
  <p>There are popular loss function as shown below:</p>
  <p>(1) L1 Loss:</p>
  <ul>
  <li>can compute the average of the sum of the absolute losses(differences) between a model's predictions and true values.</li>
  <li>'s formula is as shown below:
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/20qsszmsf0bjexmh8p16.png" alt="Image description" /></li>
  <li>'s pros are as shown below:
  <ul>
  <li>Less sensitive to outliers.</li>
  <li>We can easily complare the losses because they are just made absolute so the range of them is not big.</li>
  </ul>
  </li>
  <li>'s cons are as shown below:
  <ul>
  <li>The absolute loss of |0| cannot be differentiable according to <a href="https://math.stackexchange.com/questions/991475/why-is-the-absolute-value-function-not-differentiable-at-x-0">this post</a> and <a href="https://datascience.stackexchange.com/questions/71380/if-mean-absolute-loss-is-not-differentiable-how-it-can-be-used-in-neural-networ">this post</a>.</li>
  </ul>
  </li>
  <li>is used for a regression model.</li>
  <li>is also called Mean Absolute Error(MAE).</li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html#torch.nn.L1Loss" rel="nofollow noreferrer">L1Loss()</a> in PyTorch.</li>
  </ul>
  <p>(2) L2 Loss:</p>
  <ul>
  <li>can compute the average of the sum of the squared losses(differences) between a model's predictions and true values.</li>
  <li>'s formula is as shown below:
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63wr94q5w1dkr132hiwq.png" alt="Image description" /></li>
  <li>'s pros are as shown below:
  <ul>
  <li>All squared losses can be differentiable.</li>
  </ul>
  </li>
  <li>'s cons are as shown below:
  <ul>
  <li>Sensitive to outliers.</li>
  <li>We cannot easily complare the losses because they are squared so the range of them is big.</li>
  </ul>
  </li>
  <li>is used for a regression model.</li>
  <li>is also called Mean Squared Error(MSE).</li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss" rel="nofollow noreferrer">MSELoss()</a> in PyTorch</li>
  </ul>
  <p>(3) Huber Loss:</p>
  <ul>
  <li>can do the similar computation of either L1 Loss or L2 Loss depending on the absolute losses(differences) between a model's predictions and true values compared with <code>delta</code> which you set.
  *Memos:
  <ul>
  <li><code>delta</code> is 1.0 basically.</li>
  <li>Be careful, the computation is not exactly same as L1 Loss or L2 Loss according to the formulas below.</li>
  </ul>
  </li>
  <li>'s formula is as shown below. *The 1st one is L2 Loss-like one and the 2nd one is L1 Loss-like one:
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l90k9w1cswr768d9mzor.png" alt="Image description" /></li>
  <li>'s pros are as shown below:
  <ul>
  <li>Less sensitive to outliers.</li>
  <li>All losses can be differentiable.</li>
  <li>We can more easily complare the losses than L2 Loss because only small losses are squared so the range of them is smaller than L2 Loss.</li>
  </ul>
  </li>
  <li>'s cons are as shown below:
  <ul>
  <li>The computation is more than L1 Loss and L2 Loss because the formula is more complex than them.</li>
  </ul>
  </li>
  <li>is used for a regression model.</li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html#torch.nn.HuberLoss" rel="nofollow noreferrer">HuberLoss()</a> in PyTorch.</li>
  <li>with <code>delta</code> of 1.0 is same as Smooth L1 Loss which is <a href="https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html" rel="nofollow noreferrer">SmoothL1Loss()</a> in PyTorch.</li>
  </ul>
  <p>(4) BCE(Binary Cross Entropy) Loss:</p>
  <ul>
  <li>can compute the differences between a model's binary predictions and true binary values.</li>
  <li>s' formula is as shown below:
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jigp72l8v0qakuwvp1km.png" alt="Image description" /></li>
  <li>is used for Binary Classification. *Binary Classification is the technology to classify data into two classes.</li>
  <li>is also called Binary Cross Entropy or Log(Logarithmic) Loss.</li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss" rel="nofollow noreferrer">BCELoss()</a> in PyTorch.
  *Memos:
  <ul>
  <li>There is also <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="nofollow noreferrer">BCEWithLogitsLoss()</a> which is the combination of <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="nofollow noreferrer">BCE Loss</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html" rel="nofollow noreferrer">Sigmoid Activation Function</a> in PyTorch.</li>
  <li>Sigmoid Activation Function suits BCE Loss to be more stable.</li>
  </ul>
  </li>
  </ul>
  <p>(5) Cross Entropy Loss:</p>
  <ul>
  <li>can compute the differences(0~1) between a model's predictions and true values.</li>
  <li>s' formula is as shown below:
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xhp1a7wp6jbxwjir8l9i.png" alt="Image description" /></li>
  <li>is used for Multiclass Classification and Computer Vision.
  *Memos:
  <ul>
  <li>Multiclass Classification is the technology to classify data into multiple classes.</li>
  <li>Computer vision is the technology which enables a computer to understand objects.</li>
  </ul>
  </li>
  <li>is <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="nofollow noreferrer">CrossEntropyLoss()</a> in PyTorch.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-18T01:14:32.073Z
Answers.LastActivityDate: 2024-06-18T01:14:32.073Z
Answers.UserDisplayName: Super Kai - Kazuya Ito
Answers.Body: <p>I think you are confused about the terms <em>error</em>, <em>loss function</em> and <em>metric</em>.</p>
  <p>To summarize:</p>
  <ol>
  <li>Error (vector) tells us how off each prediction is.</li>
  <li>Loss function maps the error (vector) to a single number (loss).</li>
  <li>Loss and metric (scalar) describes overall how well the predictions are, serving as direction towards which the model should improve.</li>
  </ol>
  <p>Note these terms are general and do not apply only to neural networks.</p>
  <p><strong>TL;DR</strong></p>
  <p>Let's take a regression problem as example. Suppose we have a regression model which predicts [-4, 3, 0, 7] where the ground truth is [0, 1, 2, 3]. Clearly, some predictions are off. But by how far? Intuitively, we can get the &quot;off&quot; part by subtraction:</p>
  <p>prediction - actual = [-4, 3, 0, 7] - [0, 1, 2, 3] = <strong>[-4, 2, 2, 4]</strong></p>
  <p>which is what we call the <strong>error (vector)</strong>.</p>
  <p>Now we show the error to the model and say, &quot;hey, those are the wrongs you made, do better next round&quot;, to which the model replies, &quot;sure, which one do you prefer?&quot; and hands you 2 new predictions:</p>
  <ol>
  <li>[-2, 3, 0, 5]</li>
  <li>[0, 1, 2, 10]</li>
  </ol>
  <p>Which one is better? Well, it depends on you goal: if you want &quot;as close to the truth as possible on average&quot;, then 1) may be better; or if you say &quot;no prediction should be below the truth&quot;, then 2) is the only choice. Regardless, you have to tell the model <em>which one direction it should improve towards</em> - and this is what <strong>loss function</strong> is for.</p>
  <p>Let's say we decide to use mean-squared error (MSE) as loss function. Computing gives:</p>
  <p>Loss_1 = Mean(Square(error vector)) = Mean(Square([-2,3,0,5] - [0,1,2,3])) = Mean(Square([-2,2,-2,2])) = Mean([4,4,4,4]) = <strong>4</strong></p>
  <p>Loss_2 = Mean(Square(error vector)) = Mean(Square([0,1,2,10] - [0,1,2,3])) = Mean(Square([0,0,0,7])) = Mean([0,0,0,49]) = <strong>12.25</strong></p>
  <p>We call the result of loss function <strong>loss (scalar)</strong>. We want the loss to be small (minimize), so we tell the model, &quot;1) is the better way to go&quot;. Notice that the loss is <strong>a scalar</strong>, and is an indicator of <em>goodness of prediction</em>. In real, a model improves its prediction ('learn') by repeating</p>
  <p>make predictions -&gt; compute error vector -&gt; compute loss -&gt; feed to optimization algorithm -&gt; update model parameters -&gt; make new predictions</p>
  <p>(Often you would see the terms <em>loss</em> and <em>metric</em> used interchangeably; in brief, metric is the ultimate goal you want to achieve, a number you want to maximize; however for practical reason the optimization process prefers differentiable function, but metric may not be. In this case we usually choose a loss function which is both differentiable and 'close' to the metric. Since MSE is differentiable, it can serve as loss and metric.)</p>
  <p><strong>Update</strong></p>
  <p>In the code quoted, the computation of loss is hidden in the line <code>dW1 = E1 * A2 * (1 - A2)</code> (and similarly in the upcoming <code>E2</code> and <code>dW2</code> lines). Here, back-propagation is applied which implicitly assumes a squared loss is used. Through some maths tricks and simplification, we arrive at the <code>E1</code> formula. You may find the full derivation <a href="https://en.wikipedia.org/wiki/Backpropagation#Finding_the_derivative_of_the_error" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T04:16:59.230Z
Answers.Text: Thank you so much for your clear response, now it makes a lot more sense. So the error vector (prediction-actual) is always the same regardless of a regression and classification?. As for the loss function, this is used on the optimization algorithm (such as adam), is this correct?
Answers.UserDisplayName: user20170158
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T04:20:01.893Z
Answers.Text: 1) for the error vector - I think yes, at least no exception comes to my head. 2) yes, loss (function) is for the optimization algorithm.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T04:53:06.913Z
Answers.Text: by chance is there any online resource you can suggest to read more on the subject? every numpy neural network I can find uses the error vector as you described but I can't find any example of what they do with the loss function itself, other than just calculating it.
Answers.UserDisplayName: user20170158
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T06:27:51.430Z
Answers.Text: do you mean on e.g. why we pick one loss function other than another?
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T13:39:14.307Z
Answers.Text: No, I mean where is exactly the loss function used?, all the codes I find, they calculate the error vector and use it on the backpropagation. Not the loss.
Answers.UserDisplayName: user20170158
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-29T05:56:21.400Z
Answers.Text: can you show one example of these code? Easier to elaborate from there.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-29T22:49:09.123Z
Answers.Text: Yes, I can't paste it here (although I will edit my question with the respective code), but an example can be seen here: https://machinelearninggeek.com/backpropagation-neural-network-using-python/
  Note that he calculates the mse but as a metric only, while the error vector is calculated as pred-output, which is what's actually used in the backpropagation.
Answers.UserDisplayName: user20170158
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T10:37:25.270Z
Answers.Text: @user20170158 updated the answer.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T03:30:27.207Z
Answers.LastActivityDate: 2024-02-27T01:21:51.523Z
Answers.LastEditDate: 2024-02-27T01:21:51.523Z
Answers.UserDisplayName: lpounng
Body: <p>I've been studying NNs with tensorflow and decided to code a simple NN from scratch to get a better idea on hwo they work.</p>
  <p>It my understanding that the cost is used in backpropagation, so basically you calculate the error between prediction and actual and backpropagate from there.</p>
  <p>However, in all the examples I read online, even the ones doing classification, just use:</p>
  <p>error=actual-prediction instead of: error=mse(actual-prediction) or: error=cross_entropy(actual-prediction)</p>
  <p>And they leave mae/rmse etc just as a metric, as per my understanding (probably wrong) understanding, these should/could be used to calculate the error as well. On the other hand, while working with tensorflow, the loss function I use, does change the output and its not just a metric.</p>
  <p>What's my error in here?</p>
  <p>In other words, isn't the loss function same as the error function?</p>
  <p>Example code (taken from: machinelearninggeek.com/backpropagation-neural-network-using-python/):</p>
  <p>Note how the MSE is used as metric only, while backpropagation only uses pred-outputs. (E1 = A2 - y_train)</p>
  <pre><code>for itr in range(iterations):    
      
      # feedforward propagation
      # on hidden layer
      Z1 = np.dot(x_train, W1)
      A1 = sigmoid(Z1)
  
      # on output layer
      Z2 = np.dot(A1, W2)
      A2 = sigmoid(Z2)
      
      
      # Calculating error
      mse = mean_squared_error(A2, y_train)
      acc = accuracy(A2, y_train)
      results=results.append({&quot;mse&quot;:mse, &quot;accuracy&quot;:acc},ignore_index=True )
      
      # backpropagation
      E1 = A2 - y_train
      dW1 = E1 * A2 * (1 - A2)
  
      E2 = np.dot(dW1, W2.T)
      dW2 = E2 * A1 * (1 - A1)
  
      
      # weight updates
      W2_update = np.dot(A1.T, dW1) / N
      W1_update = np.dot(x_train.T, dW2) / N
  
      W2 = W2 - learning_rate * W2_update
      W1 = W1 - learning_rate * W1_update
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <neural-networks><machine-learning><deep-learning><tensorflow><loss>
Title: What is loss function in Neural Networks?
UserDisplayName: user20170158
ThreadText: # What is loss function in Neural Networks?
  
  (Asked by: username_0 on 2023-01-26)
  
  I've been studying NNs with tensorflow and decided to code a simple NN from scratch to get a better idea on hwo they work.
  It my understanding that the cost is used in backpropagation, so basically you calculate the error between prediction and actual and backpropagate from there.
  However, in all the examples I read online, even the ones doing classification, just use:
  error=actual-prediction instead of: error=mse(actual-prediction) or: error=cross_entropy(actual-prediction)
  And they leave mae/rmse etc just as a metric, as per my understanding (probably wrong) understanding, these should/could be used to calculate the error as well. On the other hand, while working with tensorflow, the loss function I use, does change the output and its not just a metric.
  What's my error in here?
  In other words, isn't the loss function same as the error function?
  Example code (taken from: machinelearninggeek.com/backpropagation-neural-network-using-python/):
  Note how the MSE is used as metric only, while backpropagation only uses pred-outputs. (E1 = A2 - y_train)
  for itr in range(iterations):    
      
      # feedforward propagation
      # on hidden layer
      Z1 = np.dot(x_train, W1)
      A1 = sigmoid(Z1)
  
      # on output layer
      Z2 = np.dot(A1, W2)
      A2 = sigmoid(Z2)
      
      
      # Calculating error
      mse = mean_squared_error(A2, y_train)
      acc = accuracy(A2, y_train)
      results=results.append({"mse":mse, "accuracy":acc},ignore_index=True )
      
      # backpropagation
      E1 = A2 - y_train
      dW1 = E1 * A2 * (1 - A2)
  
      E2 = np.dot(dW1, W2.T)
      dW2 = E2 * A1 * (1 - A1)
  
      
      # weight updates
      W2_update = np.dot(A1.T, dW1) / N
      W1_update = np.dot(x_train.T, dW2) / N
  
      W2 = W2 - learning_rate * W2_update
      W1 = W1 - learning_rate * W1_update
  ```
  
  **Tags:** neural-networks, machine-learning, deep-learning, tensorflow, loss
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2023-01-27. Score: 3 | Accepted answer
  
  I think you are confused about the terms *error*, *loss function* and *metric*.
  
  To summarize:
  
  1. Error (vector) tells us how off each prediction is.
  2. Loss function maps the error (vector) to a single number (loss).
  3. Loss and metric (scalar) describes overall how well the predictions are, serving as direction towards which the model should improve.
  
  Note these terms are general and do not apply only to neural networks.
  
  **TL;DR**
  
  Let's take a regression problem as example. Suppose we have a regression model which predicts [-4, 3, 0, 7] where the ground truth is [0, 1, 2, 3]. Clearly, some predictions are off. But by how far? Intuitively, we can get the "off" part by subtraction:
  
  prediction - actual = [-4, 3, 0, 7] - [0, 1, 2, 3] = **[-4, 2, 2, 4]**
  
  which is what we call the **error (vector)**.
  
  Now we show the error to the model and say, "hey, those are the wrongs you made, do better next round", to which the model replies, "sure, which one do you prefer?" and hands you 2 new predictions:
  
  1. [-2, 3, 0, 5]
  2. [0, 1, 2, 10]
  
  Which one is better? Well, it depends on you goal: if you want "as close to the truth as possible on average", then 1) may be better; or if you say "no prediction should be below the truth", then 2) is the only choice. Regardless, you have to tell the model *which one direction it should improve towards* - and this is what **loss function** is for.
  
  Let's say we decide to use mean-squared error (MSE) as loss function. Computing gives:
  
  Loss\_1 = Mean(Square(error vector)) = Mean(Square([-2,3,0,5] - [0,1,2,3])) = Mean(Square([-2,2,-2,2])) = Mean([4,4,4,4]) = **4**
  
  Loss\_2 = Mean(Square(error vector)) = Mean(Square([0,1,2,10] - [0,1,2,3])) = Mean(Square([0,0,0,7])) = Mean([0,0,0,49]) = **12.25**
  
  We call the result of loss function **loss (scalar)**. We want the loss to be small (minimize), so we tell the model, "1) is the better way to go". Notice that the loss is **a scalar**, and is an indicator of *goodness of prediction*. In real, a model improves its prediction ('learn') by repeating
  
  make predictions -> compute error vector -> compute loss -> feed to optimization algorithm -> update model parameters -> make new predictions
  
  (Often you would see the terms *loss* and *metric* used interchangeably; in brief, metric is the ultimate goal you want to achieve, a number you want to maximize; however for practical reason the optimization process prefers differentiable function, but metric may not be. In this case we usually choose a loss function which is both differentiable and 'close' to the metric. Since MSE is differentiable, it can serve as loss and metric.)
  
  **Update**
  
  In the code quoted, the computation of loss is hidden in the line `dW1 = E1 * A2 * (1 - A2)` (and similarly in the upcoming `E2` and `dW2` lines). Here, back-propagation is applied which implicitly assumes a squared loss is used. Through some maths tricks and simplification, we arrive at the `E1` formula. You may find the full derivation [here](https://en.wikipedia.org/wiki/Backpropagation#Finding_the_derivative_of_the_error).
  
  > **Comments:**
  > **username_0**: Thank you so much for your clear response, now it makes a lot more sense. So the error vector (prediction-actual) is always the same regardless of a regression and classification?. As for the loss function, this is used on the optimization algorithm (such as adam), is this correct?  
  > **username_3**: 1) for the error vector - I think yes, at least no exception comes to my head. 2) yes, loss (function) is for the optimization algorithm.  
  > **username_0**: by chance is there any online resource you can suggest to read more on the subject? every numpy neural network I can find uses the error vector as you described but I can't find any example of what they do with the loss function itself, other than just calculating it.  
  > **username_3**: do you mean on e.g. why we pick one loss function other than another?  
  > **username_0**: No, I mean where is exactly the loss function used?, all the codes I find, they calculate the error vector and use it on the backpropagation. Not the loss.  
  > **username_3**: can you show one example of these code? Easier to elaborate from there.  
  > **username_0**: Yes, I can't paste it here (although I will edit my question with the respective code), but an example can be seen here: https://machinelearninggeek.com/backpropagation-neural-network-using-python/ Note that he calculates the mse but as a metric only, while the error vector is calculated as pred-output, which is what's actually used in the backpropagation.  
  > **username_3**: @user20170158 updated the answer.  
  
  ---
  
  ### Answer by username_1 on 2023-01-26. Score: 1
  
  There are different functions to pick from for your loss.
  The most common read online is the Mean Squared Error.
  
  It calculates how far the prediction is off from the actual value and squares it. All of them are summed up.
  
  So basically the Loss-Function "generalizes" the error over every prediction. The actual error in a case is just how far off it is.
  
  > **Comments:**
  > **username_0**: In my code I have: `code` error=outputs-training_set_outputs #derivation calcs with error `code` So by your answer I understand that the MSE would be used as a metric of how well the model generalizes only?  
  > **username_1**: The loss function is a way to evaluate your whole model, yes. However I don't get what your error should be. For me it looks like your error is just your output  
  > **username_0**: if I use different loss functions in tensorflow, I do get different results, is this because the loss function is used in the optimizer? (rather than changing the error function error=y-y_hat)  
  > **username_1**: The answer above is really good. However it's natural that different Loss Functions give you different results.  You can calculate it by hand if you want, the Mean Squared Error is also often taught in school. It's totally doable. Just think of a small example, choose different functions and then calculate. You'll see easily why you'll get different results. Just keep in mind that having a smaller error is not always good. You generally want to avoid Over- and Underfitting. Also it depends on what you want to do. There is not just one solution in AI  
  
  ---
  
  ### Answer by username_2 on 2024-06-18. Score: 0
  
  A loss function is the function which can get the losses(differences) between a model's predictions and true values to evaluate how good a model is. \*Loss function is also called Cost Function or Error Function.
  
  There are popular loss function as shown below:
  
  (1) L1 Loss:
  
  * can compute the average of the sum of the absolute losses(differences) between a model's predictions and true values.
  * 's formula is as shown below:
    ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/20qsszmsf0bjexmh8p16.png)
  * 's pros are as shown below:
    + Less sensitive to outliers.
    + We can easily complare the losses because they are just made absolute so the range of them is not big.
  * 's cons are as shown below:
    + The absolute loss of |0| cannot be differentiable according to [this post](https://math.stackexchange.com/questions/991475/why-is-the-absolute-value-function-not-differentiable-at-x-0) and [this post](https://datascience.stackexchange.com/questions/71380/if-mean-absolute-loss-is-not-differentiable-how-it-can-be-used-in-neural-networ).
  * is used for a regression model.
  * is also called Mean Absolute Error(MAE).
  * is [L1Loss()](https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html#torch.nn.L1Loss) in PyTorch.
  
  (2) L2 Loss:
  
  * can compute the average of the sum of the squared losses(differences) between a model's predictions and true values.
  * 's formula is as shown below:
    ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63wr94q5w1dkr132hiwq.png)
  * 's pros are as shown below:
    + All squared losses can be differentiable.
  * 's cons are as shown below:
    + Sensitive to outliers.
    + We cannot easily complare the losses because they are squared so the range of them is big.
  * is used for a regression model.
  * is also called Mean Squared Error(MSE).
  * is [MSELoss()](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss) in PyTorch
  
  (3) Huber Loss:
  
  * can do the similar computation of either L1 Loss or L2 Loss depending on the absolute losses(differences) between a model's predictions and true values compared with `delta` which you set.
    \*Memos:
    + `delta` is 1.0 basically.
    + Be careful, the computation is not exactly same as L1 Loss or L2 Loss according to the formulas below.
  * 's formula is as shown below. \*The 1st one is L2 Loss-like one and the 2nd one is L1 Loss-like one:
    ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l90k9w1cswr768d9mzor.png)
  * 's pros are as shown below:
    + Less sensitive to outliers.
    + All losses can be differentiable.
    + We can more easily complare the losses than L2 Loss because only small losses are squared so the range of them is smaller than L2 Loss.
  * 's cons are as shown below:
    + The computation is more than L1 Loss and L2 Loss because the formula is more complex than them.
  * is used for a regression model.
  * is [HuberLoss()](https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html#torch.nn.HuberLoss) in PyTorch.
  * with `delta` of 1.0 is same as Smooth L1 Loss which is [SmoothL1Loss()](https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html) in PyTorch.
  
  (4) BCE(Binary Cross Entropy) Loss:
  
  * can compute the differences between a model's binary predictions and true binary values.
  * s' formula is as shown below:
    ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jigp72l8v0qakuwvp1km.png)
  * is used for Binary Classification. \*Binary Classification is the technology to classify data into two classes.
  * is also called Binary Cross Entropy or Log(Logarithmic) Loss.
  * is [BCELoss()](https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss) in PyTorch.
    \*Memos:
    + There is also [BCEWithLogitsLoss()](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html) which is the combination of [BCE Loss](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html) and [Sigmoid Activation Function](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) in PyTorch.
    + Sigmoid Activation Function suits BCE Loss to be more stable.
  
  (5) Cross Entropy Loss:
  
  * can compute the differences(0~1) between a model's predictions and true values.
  * s' formula is as shown below:
    ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xhp1a7wp6jbxwjir8l9i.png)
  * is used for Multiclass Classification and Computer Vision.
    \*Memos:
    + Multiclass Classification is the technology to classify data into multiple classes.
    + Computer vision is the technology which enables a computer to understand objects.
  * is [CrossEntropyLoss()](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) in PyTorch.
  
  
  ---
  
  