Apologies if the answer is silly, I’ve been using BERT and not GPT2.
I think your twice-trained model would probably remember at least some of the astronomy training, as well as the physics training.
If you had a really huge corpus of physics texts it might overwrite your astronomy training, but I think it is unlikely.   Some researchers have shown that many transformers models have a lot more capacity than they need.  Also, there is probably overlap of physics and astronomy vocab.
When you fine-tune, you can define whether you want the model layers to be altered or frozen.   You could consider gradual unfreezing of layers.