It depends on the setup. Sometimes in action recognition the video sequence has one label, i.e. multiple frames have are associated with one label.
Usually in object detection there can be multiple boxes in each frame. So yes, that’s correct.