Solved by ptrblck in post #2 
              
              
                Maybe because the layers 1 and 2 are nn.ReLU() and nn.MaxPool2d. 
Not easy to tell without a model definition, but this might be the case.