Unfortunately, I can’t comment on the mismatch of the extracted frames, but don’t think it’s an “OpenCV vs. FFmpeg” issue, since OpenCV can use FFmpeg 1.
That being said, it seems as if torchvision.io uses pyav as the default reader (line of code 8), which wraps FFmpeg as seen here 5.