Input and output have size n x 21 x 512 x 512, and I pad the image to get the correct sizes.
I can go with n = 1 if needed, however, I prefer to keep > 1. This should be a minor problem since it can be parallelized.
The problem is that the number of input windows (n x 512^2 x k^2) generated by the mini-batch is too large and cannot fit into memory. With n = 1 and k = 255 (my max kernel size) and working with float32, the kernel windows occupy ~63GB of memory.