2 Random ideas.
Have you checked hidden state? Perhaps you are reinitilizing it each time you feed a sample.
Try to check if it behaves like that with other weights. If so, it’s probably a bug in your code rather than wrong validation.
Are the outputs of the CNN different? so that the problem happens in the RNN