Hey,
I am working on the same type of problem, I implemented a similar model for time series prediction and it seems the loss fluctuates a lot.
I am still debugging my code, but in the meantime did you figure out what was the issue ?