On the question of detecting where a user was most interested in over a long video, one signal could be if user replayed any particular segment of the video. This might be to understand that segment better (shows user interest in that particular segment).