@chenyuntc I dont see any improvement with changing sum to mean 
And about changing the learning rate: Changing the hyperparameter will definitely improve the performance coz my NN was still learning. But my problem is not with my networks isolated performance. Numpy give me a loss of 0.009 and acuracy more than 0.98 with the same hyperparameter i used for pytorch. But pytorch is nowhere near.