One of the possible way is usage of dense and sparse sampling.
It means collect densely from the high score part and, in opposite, collect sparsely from the low score part.
However, this method only can be applied in training.
In CVPR (perhap '15 or '16), MS utilized both single frame and sequential frames to detect the highlight in a video. I recommend you look up that paper.