@rgwatwormhill,  I just fine tuned the model and did not alter it.
If the name of that model class was MyGPT2Model and I saved it using Trainer, to load it would simply be the line MyGPT2Model.from_pretrained('/path/to/model/')?
And since it inherits from  GPT2LMHeadModel I can call the generate method to generate text like:
import MyGPT2Model
from transformers import GPT2Tokenizer

prompt = 'Huggingface Transformers is fantastic!'
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
encoded_results = tokenizer(prompt, return_tensors='pt')
tok_ids = encoded_results['input_ids']

my_model = MyGPT2Model.from_pretrained('/path/to/model/')
gen_ids = my_model.generate(tok_ids)
decoded_str = tokenizer.decode(gen_ids)

Is this correct?