Thank you very much for the feedback everyone! I’m new to huggingface transformers and pytorch in general so I apologize for the naive questions. I’ve written up some code that I think might accomplish what I am seeking to do (except for the training loop):
from transformers import GPT2LMHeadModel, AdamW
from transformers.modeling_outputs import CausalLMOutputWithPast

class GPT2Finetuned(GPT2LMHeadModel):
    def __init__(self):
        super().from_pretrained('gpt2')

    def forward(
        self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        use_cache=None,
        output_attentions=None,
        output_hidden_states=None,
        return_tuple=None,
    ):
        return_tuple = return_tuple if return_tuple is not None else self.config.use_return_tuple

        transformer_outputs = self.transformer(
            input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_tuple=return_tuple,
        )
        hidden_states = transformer_outputs[0]
        lm_logits = self.lm_head(hidden_states)
        loss = None

        #HERE I WOULD CALCULATE THE LOSS BETWEEN THE LM_LOGITS AND MY NGRAMS MODEL??

        return CausalLMOutputWithPast(
            loss=loss,
            logits=lm_logits,
            past_key_values=transformer_outputs.past_key_values,
            hidden_states=transformer_outputs.hidden_states,
            attentions=transformer_outputs.attentions,
        )


model = GPT2Finetuned()
model.train()
optimizer = AdamW(model.parameters(), lr=1e-3)
for step in range(100):
    ????

That’s about as far as I got from looking at the different examples and the GPT2LMHeadModel class. What else needs to be added/changed in order to finetune GPT2 as described above?