I think that my personal problem is that the semantics of voting are usually more related to popular/unpopular instead of quality/poor quality. Not that I have a particular problem with popular/unpopular as the semantics per se, just that that seems to be the way the community uses them (myself included most of the time).