I am not entirely sure why but I found the results more jittery using p5.js vs direct, especially on the edges of the video, so I ended up dropping it for my project and going direct. Could be related to this point @Jason mentioned?

Also it seems P5 does some rendering magic of its own - it draws the video frame to canvas and then the dots on top of that, which is not terribly efficient as you are sampling video frame twice - you can just absolute position canvas on top of video element and draw the circles only on top of the already playing video saving you pushing twice the number of video pixel data around and only needing to worry about rendering dots to canvas based on the rendered size of the canvas.