It certainly seems worth a try; the obvious problem being that you might have it learning the noise as a differentiating measure, so if you got poor-quality low-speed poses or high-quality fast poses it might have trouble learning them.
It sounds like you might want to try processing the images in both datasets first to equalise features like brightness etc.?