It all really depends on what you want to achieve. If you think that you need the entire image, you could just scale the image down using transforms.Scale(image_size) instead. But it might be the case that getting random crops helps in training. Best way would be to try both and see what each network learns.
If you’re loading images that large and you plan on just scaling them down, I would recommend preprocessing them all first (create another dataset where all the images are of the size used by the network) as it would greatly speed up training.
Good luck!