In case you have binary states, 0 or 1, network will learn to map into those values by using BCEwithlogits if your GT is binary.
Obviously initial values will tend to be 0.5, but it should become “more” binary through epochs. Don’t underestimate the loss you use, it’s very important.
What are you trying to predict? Classification it seems. I would fit to Cross entropy. Be sure it’s properly done. Then you can try to reduce frame rate. Are you using a pre-trained CNN (would be a nice idea).