Alternatively, I am down sampling the images with ConvTranspose2d itself, by choosing a stride of 2 every time. Please let me know if that is not a good idea.
At the end I will need to get the image with a specific size. What is a good operation for that?