When this issue 35 is resolved in torchvision another solution to this problem may be to use torchvision.transforms.RandomCrop on both targets and inputs which should behave over the epochs as if you would train patches (maybe this approach would even be superior as it takes patches from different places).
To crop MRI and US to the same size (so we already removed most area with zero data) you can use: https://gist.github.com/bodokaiser/b5697f95336be11aa09f6e246901e0d4 134