I think the effects you are seeing would be expected as your model was trained with “real” or clean frames to predict the next frame while you are now feeding the predictions into the model to create new frames, which might diverge a bit in each step further from the real distribution.
I haven’t tried this approach in image generation, but I think that teacher forcing during training might be a valid experiment to run. It is (or was?) used in language models, which were suffering from similar effects, so could also work for your model.
To do so, you would also use the predicted frames of the model during training and feed a real image once in a while (I think you should play around with the frequency a bit, but refer to known implementations to check what worked before) to stabilize the training.