What I would start with is one video loaded into a matrix of shape (num frames, num channels, height, width) as X and an array or size (num frames) with a 1 if there was a click on that frame and a 0 if ther was not.
From there I would follow any pytorch resnet tutorial and use one of their retrained models with the number of output classes equal to 2.