It would certainly be a nice to have, but it’s actually quite challenging to do this right and do it efficiently so you’re not spinning 16 cores at 100% feeding one model and consuming 100s of GB of memory. I have a lot of experience storing, streaming video with other synchronized streams in a past life and it’s a rather big pain in the ass to say the least.
You would ideally want to leverage the video decoders on the Nvidia card, and even more ideally keep the decoded buffers on the card, a zero-copy to Tensors ideally. It looks like Nvidia has done the basics of something like that with their DeepStream SDK but I have’t looked closely enough to see the details, they may have made shortcuts given that they claim to be feeding standard image models that don’t have a time axis.
Aside from leveraging hardware decoding, the timestamping of video, and corresponding streams that you may want to keep in synch such as audio, text, other metadata can be quite challenging. At a codec level, advance video codecs have their frames out of order, the amount of video you have to decode to get a sequence of frames in the correct order varies depending on codec and chosen encode parameters. It can be quite significant which could cause problems with buffering and memory usage. Most surveillance systems get around this by enforcing maximum keyframe intervals based on a users desired latency to storage usage tradeoff, but typical media from streaming sources or disks like blu-rays, Netflix, etc have very long keyframe intervals to reduce bandwidth since random access is minimal and people are okay waiting a few seconds for the stream to resume after changing position. If you are doing this constantly to get fixed time windows, it would be a huge overhead.
Then you’d have to figure out how to represent your targets, what are they? how are they represented relative to a stream of images with a time axis. I assume there would be a lot of possible variety and different wants here.
And I thought about all this because I looked at doing it, but it will have to wait until a possible project is viable that requires this as it is a significant undertaking