Noted that…thanks!!
I tried xavier_normal initialization with some set of parameters of bs, lr etc as mentioned above and it worked well. Now I want to try with normal initlalization and same set of parameters.
I am trying it out as
best_net = Net(conv1_kernel_size=3, conv1_output_channels=24, conv2_kernel_size=5, conv2_output_channels=24, fc1_output_size=200)
best_net.apply(init_weight_normal)

where init_weight_normal is a function
def init_weight_normal(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1 or classname.find('Linear') != -1:
        torch.nn.init.normal_(m.weight)
        m.bias.data.fill_(0.1)         

And in the main loop for each iteration, I am calling best_net.apply(init_weight_normal)
i.e. just resetting the network to normal initialization, which worked well in case of xavier normal.
But now, I am observing that for every set of parameters, the loss is steady at 2.303 and test accuracy is fixed at 10% for all epochs and in some abrupt run at some epoch loss going to some 9999543434.0 etc.
I am not able to deduce what’s going wrong in case of  normal initialization.