This happens to me all the time. So annoying... feels like something to do with how it locks the scroll direction when you first start the gesture. So even a perfectly vertical scroll will result in a back if the initial jitter has locked a horizontal motion.