I am doing some experiments to figure out the root cause. Its happening even in case of SGD optmizer with no momentum.
optimizer = optim.SGD(net.parameters(), lr=0.001)
In the following image, the slight loss jump after 2000 iterations, is after the training is continued from a checkpoint.