Ok, using
torchvision.set_video_backend('video_reader')
solved my issue for the “sec” unit. I now have [251, …], [2, 441344].
And the “pts” unit seems to be depreciated so it’s not important.
Seems to be solved.
Though it says in the documentation I should compile torchvision from source but I didn’t and it still works, maybe something will be wrong later