It all depends on how much data you have and on how long you train. If you have billions of texts and train for billions of steps, the astronomy part will hardly be remembered. What you can do, however, is merging them and shuffling the data so that the model evenly sees both datasets at the same time.
PS it is not necessarily the case that only the last layer is finetuned. This is something that you as a user can decide.