Answers.Body: <p>Yes, that is completely possible. Instead of using a mean of loss weights, you can use higher weights for tokens of choice and calculate weighted mean.</p>
  <p>Pytorch code for the same <a href="https://discuss.pytorch.org/t/using-nllloss-weighted-loss-how-to-define-a-good-loss/39263/5" rel="nofollow noreferrer">Code Source</a></p>
  <pre><code>log_prob = torch.tensor([[-0.0141, -4.2669],
                       [-0.0141, -4.2669]])
  target = torch.tensor([0, 1])
  weight = torch.tensor([2.0, 3.0])
  criterion = nn.NLLLoss()
  criterion_weighted = nn.NLLLoss(weight=weight)
  
  print(criterion(log_prob, target))
  &gt; tensor(2.1405)
  print(criterion_weighted(log_prob, target))
  &gt; tensor(2.5658)
  </code></pre>
  <p>The weights will enforce/bias your transformer's attention mechanism to focus on specific relationships in the data. Note the weights will be a hyper parameter you need to fine tune.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-11T10:33:38.427Z
Answers.LastActivityDate: 2023-10-11T10:33:38.427Z
Answers.UserDisplayName: kaiser
Body: <p>I'm developing an encoder-decoder based transformer model and I would like to ask if there are ways to incentivize or penalize certain tokens during training.</p>
  <p>I'm working on a translation task where the encoder input must be decoded into its proper product name. I have labels such as brand, name, and unit of measure, etc which are available during training but not on inference.</p>
  <p>Currently when predicting the brand portion (which usually appears early in the sequence) of the output, the heatmap shows that it does not give focus to the latter part of the encoder which produce an output that the brand and product name, and unit of measure does not belong to each other.</p>
  <p>I was thinking if there's a way to force the transformer during training to give more weight to different token types other that its own.</p>
  <p>For example:</p>
  <ol>
  <li>Brand tokens (decoder) should give more weight to name tokens (encoder) than other brand tokens (encoder)</li>
  <li>Name tokens (decoder) should give more to brand token (encoder) and unit of measure token (encoder)</li>
  </ol>
  
Tags: <neural-networks><machine-learning><deep-learning><transformer><pytorch>
Title: How to force Transformer to give more weight to certain tokens
UserDisplayName: jasperagrante
ThreadText: # How to force Transformer to give more weight to certain tokens
  
  (Asked by: username_0 on 2023-10-04)
  
  I'm developing an encoder-decoder based transformer model and I would like to ask if there are ways to incentivize or penalize certain tokens during training.
  
  I'm working on a translation task where the encoder input must be decoded into its proper product name. I have labels such as brand, name, and unit of measure, etc which are available during training but not on inference.
  
  Currently when predicting the brand portion (which usually appears early in the sequence) of the output, the heatmap shows that it does not give focus to the latter part of the encoder which produce an output that the brand and product name, and unit of measure does not belong to each other.
  
  I was thinking if there's a way to force the transformer during training to give more weight to different token types other that its own.
  
  For example:
  
  1. Brand tokens (decoder) should give more weight to name tokens (encoder) than other brand tokens (encoder)
  2. Name tokens (decoder) should give more to brand token (encoder) and unit of measure token (encoder)
  
  **Tags:** neural-networks, machine-learning, deep-learning, transformer, pytorch
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-11. Score: 0
  
  Yes, that is completely possible. Instead of using a mean of loss weights, you can use higher weights for tokens of choice and calculate weighted mean.
  
  Pytorch code for the same [Code Source](https://discuss.pytorch.org/t/using-nllloss-weighted-loss-how-to-define-a-good-loss/39263/5)
  
  ```
  log_prob = torch.tensor([[-0.0141, -4.2669],
                       [-0.0141, -4.2669]])
  target = torch.tensor([0, 1])
  weight = torch.tensor([2.0, 3.0])
  criterion = nn.NLLLoss()
  criterion_weighted = nn.NLLLoss(weight=weight)
  
  print(criterion(log_prob, target))
  > tensor(2.1405)
  print(criterion_weighted(log_prob, target))
  > tensor(2.5658)
  
  ```
  
  The weights will enforce/bias your transformer's attention mechanism to focus on specific relationships in the data. Note the weights will be a hyper parameter you need to fine tune.
  
  
  ---
  
  