So my friend helped me find the culprit. the problem arises due to batchnorm layer. the batch stats are used in training but the whole dataset stat is used while validation and this is calculated during training. hence the forward pass of second dataset messes with the dataset stats calculated.
This blog article should be helpful(tough its written for tf I think the problem is same): https://towardsdatascience.com/pitfalls-of-batch-norm-in-tensorflow-and-sanity-checks-for-training-networks-e86c207548c8