Solved by ptrblck in post #2 
              
              
                I’ve tested your code for some random input and the model could fit it. 
Using CIFAR10, I just set the channels to 6 and  12 in the conv layers. 
The model indeed struggles to learn and the default random init seems to be better. 
Changing fan_in = shape[1] for linear layers makes it a bit better. 
…