I have a dataset of scientific abstracts that I would like to use to finetune GPT2. However, I want to use a loss between the output of GPT2 and an N-grams model I have to adjust the weights. Is it possible to do this using huggingface transformers and if so, how? Thank you in advance!
EDIT:
Let me be a little more explicit. I would like to take the base gpt2 model and finetune it for text generation on my dataset of scientific abstracts. However, I would like to replace the loss function that the base gpt2 uses for my own that is based off an N-grams model I have. Ultimately, I would like for the finetuned model to generate scientific-sounding abstracts of a given length based off an initial sentence or two.