I’m not sure exactly where your model is going off, but you can find a more standard approach for working with mnist at https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py if that is helpful for your use case.  One thing that looks interesting to me in your model is that you don’t have activations between your convolutional layers in your Blocks, but I don’t know if that alone would cause it to not learn anything.