I suspect your single downvote would be have been countered by several people supplying corrective upvotes, so they could have ended up with more upvotes than they would have done without the initial downvote.