Sally:

And I think it is weird that actually the video has different frames and each frame are having different content, the extracted features should not be that similar. Or am I wrong with this concept?


It depends, what the input frames are showing and what the model predicts.
E.g. if the model predicts the same class for all input frames, I would assume that the extracted features are quite equal.
Depending on your current use case and input, the model might predict a single random class or just completely random output, since it wasn’t trained on your dataset.
Did you check, what the model prediction is for these frames? If the prediction is equally wrong, the model isn’t able to extract useful features from your frames.