Solved by ptrblck in post #7 
              
              
                Well, as you can see the network architecture in Keras is quite different from the example in Pytorch. Anyway, lower the learning rate to 0.001 and you should be fine with batch_size=1.