Input tensors of [5000, 5000] are quite huge. How much memory was used for a single sample of your [1024, 1024] input? Note that the [5000, 5000] has approx. 24x more elements.
Also, have a look at torch.utils.checkpoint to trade compute for memory.