I tried different data patterns and the problem didn’t occur anymore. I also tried to clip the gradient in the RNN and found that it worked. So I guess it is probably a explosive gradient problem, but I couldn’t figure out why only the valid loss is NaN…