Answers.Body: <p>The general approach is to use <em>sample weights</em> to weight the loss function: for a sample <span class="math-container">$i$</span> in the dataset, you weight its loss <span class="math-container">$l_i$</span> by <span class="math-container">$w_i$</span>. In this way you can make an error bigger if the sample is more important (hoping the network to better predict it), or smaller if the sample is less important. The drawback of such approach is that you need to have an extra column of file along your dataset to store the sample weights, which are sample-specific, apart from being able to define them at all.</p>
  <p>Other approaches involve:</p>
  <ul>
  <li><em>Class weights</em> specific for classification. Basically is like applying sample weights, where the weights for samples belonging to the same class are the same.</li>
  <li>Otherwise you can <em>implicitly</em> weight a sample by sampling it more frequently in a mini-batch, or over-sampling (duplicating) it: in this way, a data point <span class="math-container">$x$</span> will be present in more batches, and, in principle, you can also include copies of <span class="math-container">$x$</span> in the same batch to increase the overall loss. Similarly, you can under-sample or sample <span class="math-container">$x$</span> less frequently to decrease its importance.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T23:55:51.213Z
Answers.Text: Having an additional column for weights and using them in the loss function is exactly what I was thinking about. Is there an out-of-the-box way of doing it, for example, in TensorFlow? Or do I need to inject my own custom loss function? Much appreciated.
Answers.UserDisplayName: PlinyTheElder
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-18T13:21:22.197Z
Answers.Text: @PlinyTheElder `tf.keras` supports sample weights by default. You can pass them in `model.fit(..., sample_weight=dataset['sample_w_col'])`. If you have a custom model and `train_step(batch)`, you still pass them in `model.fit` but remember to unpack the batch, like `x, y, w = batch` and then apply them to the loss, e.g, `self.compiled_loss(y_pred, y, sample_weight=w)`, where compiled loss is what you put in `model.compile`. The metrics also follow a similar pattern, since these can be weighted to.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-18T15:10:48.577Z
Answers.Text: Perfect - thanks a million. This is exactly what I was looking for.
Answers.UserDisplayName: PlinyTheElder
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T13:46:22.077Z
Answers.LastActivityDate: 2023-07-17T13:46:22.077Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In other words, is it possible to assign &quot;weights&quot; to data points during model training? Is there a standard technique for it?</p>
  <p>It seems like it the math would be straightforward enough for this functionality to be included into DNN frameworks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-17T11:28:23.633Z
Comments.Text: Do all data points belong to the same class (if its classification) ? If so you can weight the classes directly. Otherwise you will probably need to save the weight for each data sample and use it when you compute the loss.
Comments.UserDisplayName: Lelouch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-18T00:01:22.287Z
Comments.Text: @Lelouch: The model is expected to predict a set of float numbers based on another set of float numbers. Do you know if there's a good example somewhere on how to write such a custom loss function using, for example, TensorFlow?
Comments.UserDisplayName: PlinyTheElder
Tags: <machine-learning><deep-neural-networks><data-preprocessing>
Title: When training a DNN model, how can I make some training data points more important than others?
UserDisplayName: PlinyTheElder
ThreadText: # When training a DNN model, how can I make some training data points more important than others?
  
  (Asked by: username_0 on 2023-07-16)
  
  In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it?
  
  It seems like it the math would be straightforward enough for this functionality to be included into DNN frameworks.
  
  **Tags:** machine-learning, deep-neural-networks, data-preprocessing
  
  > **Comments:**
  > **username_1**: Do all data points belong to the same class (if its classification) ? If so you can weight the classes directly. Otherwise you will probably need to save the weight for each data sample and use it when you compute the loss.  
  > **username_0**: @username_1: The model is expected to predict a set of float numbers based on another set of float numbers. Do you know if there's a good example somewhere on how to write such a custom loss function using, for example, TensorFlow?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-17. Score: 2 | Accepted answer
  
  The general approach is to use *sample weights* to weight the loss function: for a sample $i$ in the dataset, you weight its loss $l\_i$ by $w\_i$. In this way you can make an error bigger if the sample is more important (hoping the network to better predict it), or smaller if the sample is less important. The drawback of such approach is that you need to have an extra column of file along your dataset to store the sample weights, which are sample-specific, apart from being able to define them at all.
  
  Other approaches involve:
  
  * *Class weights* specific for classification. Basically is like applying sample weights, where the weights for samples belonging to the same class are the same.
  * Otherwise you can *implicitly* weight a sample by sampling it more frequently in a mini-batch, or over-sampling (duplicating) it: in this way, a data point $x$ will be present in more batches, and, in principle, you can also include copies of $x$ in the same batch to increase the overall loss. Similarly, you can under-sample or sample $x$ less frequently to decrease its importance.
  
  > **Comments:**
  > **username_0**: Having an additional column for weights and using them in the loss function is exactly what I was thinking about. Is there an out-of-the-box way of doing it, for example, in TensorFlow? Or do I need to inject my own custom loss function? Much appreciated.  
  > **username_2**: @username_0 `tf.keras` supports sample weights by default. You can pass them in `model.fit(..., sample_weight=dataset['sample_w_col'])`. If you have a custom model and `train_step(batch)`, you still pass them in `model.fit` but remember to unpack the batch, like `x, y, w = batch` and then apply them to the loss, e.g, `self.compiled_loss(y_pred, y, sample_weight=w)`, where compiled loss is what you put in `model.compile`. The metrics also follow a similar pattern, since these can be weighted to.  
  > **username_0**: Perfect - thanks a million. This is exactly what I was looking for.  
  
  ---
  
  