Solved by futscdav in post #4 
              
              
                The adaptive pooling only happens right before the classifier part of the network and just serves to average the size down to a single feature vector. Issue with something like 28x28 is that since there are several layers with stride=2, you will be disproportionally losing values out of the right si…