Solved by ptrblck in post #4 
              
              
                An input of [batch_size, 1024, 1, 1] won’t work for a standard resnet, as the conv layers use kernels with more than a single pixel and you are also using pooling layers, which won’t be able to decrease the spatial dimensions. 
You could try to 

reshape the encoder output, and create “fake” spatial…