I agree that removing the points may cause the issue where someone is upvoted tremendously without merit.  After all, I often decide whether to upvote a comment based on its current score.  If I see someone with a score of 20 for a reasonable comment yet someone with a score of 1 for a brilliant comment, I would be more likely to upvote the lower score.  I think this would cause a bit of Karma inflation if the points were not visible.