My plan for dealing with downvoting is just not to display points on comments any more.  The hard part is that to make that work, I also have to come up with alternative ways for people to find the comments they're most interested in on big threads.