Change gpt2 language model loss function
Hi all,

I want to include a new loss term for the gpt2 training loss. I am using the script run_lm_finetuning from the examples. This is my command:

python examples/run_lm_finetuning.py --output_dir=output --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=$TRAIN_FILE --eval_data_file=$TEST_FILE --overwrite_output_dir --max_steps 50


 but I really can't figure out which loss function is being used. If i print inside the GPT2LMHeadModel forward function, nothing happens. Could you please tell me which loss function should I change?

Thank you a lot.
Afaik the "default" loss function that gets computed if you pass your labels to `GPT2LMHeadModel` is `torch.nn.CrossEntropyLoss`. If you want to use a different loss function, can't you just grab the logits from the model and apply your own?

Source:
https://github.com/huggingface/transformers/blob/391db836ab7ed2ca61c51a7cf1b135b6ab92be58/transformers/modeling_gpt2.py#L539<|||||>Unfortunately if I print a string inside the forward function, and then I run the training script, I don't get anything printed, so it seems like the training script is not using that function at all.<|||||>Hello! The `GPT2LMHeadModel` does have a way to compute its own cross-entropy loss, but only when the `labels` are specified -> you're providing the values like so:
```
model(inputs, labels=inputs)
```
and the model takes care of shifting the inputs to calculate a causal language modeling loss on them with cross-entropy.

If you wish to use your own loss function, don't specify the labels and the model will return a tuple containing the language modeling logits as the first value.<|||||>Hi,

thanks for your answer.
Can you tell me why a print statement inside the forward fuction of the GPT2LMHeadModel doesn't print anything when I run the run_lm_finetuning script ?

Which is the forward function I need to change?

Thanks.<|||||>Where have you put your print statement? Do you have `transformers` installed in your environment or is it relying on the cloned repository? You could try to add a breakpoint and debug it to see which function calls are made and how the loss is calculated.

Once again, if you wish to use your own loss function, don't specify the labels and the model will return a tuple containing the language modeling logits as the first value.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
