<p>Actually I encountered this issue myself.</p>
<p>My solution was to wrap the model in my own class.
I then overrode the forward method. ( I was using pytorch lightning)</p>
<pre><code>class WrapModel(pl.LightningModule):
    self.model = T5ForConditionalGeneration...

def forward(self, **inputs):
    outputs = self.model(**inputs)
    loss = new_loss_logic(outputs.specific_hidden_state)
    outputs.loss = loss
</code></pre>
<p>I am relatively new to the field, but I think this should work.
If you wish to stay within HF, you can always make the wrapping class inherit the T5Model, then the call to the underlying <code>forward</code> is <code>self(**inputs)</code> or can be down with <code>super()</code>, but the rest remains the same</p>

CC BY-SA 4.0
2023-03-14T19:29:15.853Z
2023-03-14T22:50:00.007Z
2023-03-14T22:50:00.007Z
Thomas K
<p>Currently I am finetuning <code>transformers</code> T5 model for translation task. As part of the dataset, I am given sentences in Japanese, their translation to English, and for every English sentence I am also given a few (English) words which need to be in the translated sentence.</p>
<p>Is there a way to modify the <code>Seq2SeqTrainer</code> and the loss being used in order to penalize the model (while training) for not including the given words in the translated sentence?</p>

<transformer><machine-translation><fine-tuning>
Fine-Tuning T5 with specific penalty
Dani
# Fine-Tuning T5 with specific penalty

(Asked by: username_0 on 2023-03-14)

Currently I am finetuning `transformers` T5 model for translation task. As part of the dataset, I am given sentences in Japanese, their translation to English, and for every English sentence I am also given a few (English) words which need to be in the translated sentence.

Is there a way to modify the `Seq2SeqTrainer` and the loss being used in order to penalize the model (while training) for not including the given words in the translated sentence?

**Tags:** transformer, machine-translation, fine-tuning


---

## 1 answers

### Answer by username_1 on 2023-03-14. Score: 1

Actually I encountered this issue myself.
My solution was to wrap the model in my own class.
I then overrode the forward method. ( I was using pytorch lightning)
class WrapModel(pl.LightningModule):
    self.model = T5ForConditionalGeneration...

def forward(self, **inputs):
    outputs = self.model(**inputs)
    loss = new_loss_logic(outputs.specific_hidden_state)
    outputs.loss = loss

I am relatively new to the field, but I think this should work.
If you wish to stay within HF, you can always make the wrapping class inherit the T5Model, then the call to the underlying forward is self(**inputs) or can be down with super(), but the rest remains the same


---

