Soumyajit_Das:

Also one fact, I do not understand, if the training gets better, that means the loss is getting reduced and accordingly the network parameters(weights) are updated, why is the network giving less Test Accuracy on Test Dataset .


This is called overfitting and your model starts to learn training set specific features, which do not generalize well to unseen examples.
These features depend on the dataset and you can generally counter this effect by used regularization or increasing data augmentation.
A simple example would be a face classifier:
While the model should learn that a face contains a nose, eyes, mouth, etc., it might overfit to specific features like blue eyes, black hair etc. to detect a face.