Hi pal, could I know if you still have this issue? Any hint to solve it? I run into this problem lately and kinda stuck here.  I am thinking if it is caused by gradient vanishing and try to solve it by adding the layer normalization. I am still monitoring the process, see if it will go well.