1024 is pretty large for this kind of model.
In the original paper the authors also used a sliding window approach to get their model working.
I tried to debug the model as @vmirly1 suggested, however, there seem to be a few issues in your code.
If I resize the images to 512 or 256, I get a size mismatch error even if I use crop=True in the crop_concat layers. I fixed it with an ugly hack to get it working.
However, I also realized that you are actually never using the concatenated output of the crop_conat layers.
If you assign the result back to x, you’ll get another error, since the number on input channels won’t match for your conv layers.
If I use my implementation 39 as a drop-in replacement for your model, I can get the model working with a batch size of 1 for the original image shapes using approx. 7GB of GPU memory.
Note that the implementations differ a bit, e.g. your model seem to have an additional layer in the bottleneck, so that will most likely explain the difference in memory usage.