Yes it is super weird. I have tried removing the gradient clip. It did not make a difference.
But if there was an issue with the gradient clip, the same effect should be there for the cpu version right?