Just to make sure I understand the issue correctly:
You are feeding two images simultaneously (both are quite large at ~2000x2000 pixels).
While training the model from scratch everything works fine.
If you want to fine-tune a pretrained model, you are running out of memory.
Is this correct or do I miss something?