I change my learning rate from 1e-3 to 1e-5, and trained it again for 200 epochs, I am getting a better plot now,
Screenshot_2019-10-06 Google Colaboratory(2).png1197×602 75.2 KB

The loss is decreasing, till i guess 120-130 epochs? and then it kinda starts increasing, any possible explanation for this?