I was thinking about trying this strategy out, but I would have to figure out how to tell the cuda kernels what the actual image size is. E.g. if I have an image of size 137x200 but I want to embed it into a square 200x200 (and then insert it as image k in the image tensor) then I want to let the kernel know that image is only 137x200, to avoid having the net learn this buffer of zeros. It turns out if you try and build an autoencoder this way, it implicitly learns the shape and places images of a similar pixel size next to each other, which is not really what you want.