@ptrblck thanks for the reply. The issue was resolved but I observe weird accuracy drop after certain round. Please see the attached pic.

fig720×576 26.8 KB

State of accuracy loss is as follow:
Round 187: accuracy 0.66 loss : 0.018870125
Round 188: accuracy 0.1525 loss: 12.478773
Round 289: accuracy 0.68 loss: 0.014562485
Round 290: accuracy 0.3 loss : 5.9329453

Weights and gradients are being updated normal. How can we debug this? Do I need to change model? or something else?
EDIT
The above plot was built using the learning rate 0.0003. I have checked if I use learning rate of 0.0001 then accuracy goes up by 0.67 instead of dropping.