I had included the zero inititialization for decoder inside the loop - zeroing it for every step. I have corrected it. The training is occuring but at extremely slow pace. A specific question would be, where to look to know why training is too slow? Any general suggestions would be helpful.
train(4000, model, train_batches)

epoch: 0 -- loss: 4.162579536437988
epoch: 100 -- loss: 4.112975120544434
epoch: 200 -- loss: 4.112966060638428
epoch: 300 -- loss: 4.112963676452637
epoch: 400 -- loss: 4.1129631996154785
epoch: 500 -- loss: 4.1129631996154785
epoch: 600 -- loss: 4.11296272277832
epoch: 700 -- loss: 4.112962245941162
epoch: 800 -- loss: 4.112962245941162
epoch: 900 -- loss: 4.112961769104004
epoch: 1000 -- loss: 4.112961769104004
epoch: 1100 -- loss: 4.112961292266846
epoch: 1200 -- loss: 4.112961292266846
epoch: 1300 -- loss: 4.112961292266846
epoch: 1400 -- loss: 4.112961292266846
epoch: 1500 -- loss: 4.1129608154296875
epoch: 1600 -- loss: 4.1129608154296875
epoch: 1700 -- loss: 4.1129608154296875
epoch: 1800 -- loss: 4.1129608154296875
epoch: 1900 -- loss: 4.1129608154296875
epoch: 2000 -- loss: 4.112960338592529
epoch: 2100 -- loss: 4.112960338592529
epoch: 2200 -- loss: 4.112960338592529
epoch: 2300 -- loss: 4.112959861755371
epoch: 2400 -- loss: 4.112959861755371
epoch: 2500 -- loss: 4.112959384918213
epoch: 2600 -- loss: 4.112959384918213
epoch: 2700 -- loss: 4.112958908081055
epoch: 2800 -- loss: 4.112958908081055
epoch: 2900 -- loss: 4.1129584312438965
epoch: 3000 -- loss: 4.1129584312438965
epoch: 3100 -- loss: 4.112957954406738
epoch: 3200 -- loss: 4.112957954406738
epoch: 3300 -- loss: 4.112957954406738
epoch: 3400 -- loss: 4.112957954406738
epoch: 3500 -- loss: 4.11295747756958
epoch: 3600 -- loss: 4.11295747756958
epoch: 3700 -- loss: 4.11295747756958
epoch: 3800 -- loss: 4.112957000732422
epoch: 3900 -- loss: 4.112957000732422