I was able to do this perfectly. I was making very basic mistake in calculating loss. Instead of
loss = criterion(outputs, torch.tensor(labels))
I needed to use
loss = criterion(outputs[:,0], torch.tensor(labels))
and the model works perfectly. Infact GRU does it very well. Anyways thanks for trying to help me.