Solved by ptrblck in post #6 
              
              
                The output sizes are beautifully explained in Stanford’s CS231n. 
In my architecture each conv layer will reduce the spatial size by two in both dimensions, while the pooling layers will divide them by two. 
For an input size of [batch_size, 3, 512, 512] you will end up as: 
conv1: 512 - 2 = 510
poo…