Relying on the face might be human nature - even babies prefer to look at attractive people. But, of course, judging someone based on the geometry of his features is, from a moral and legal standpoint, no better than judging him based on the color of his skin.
Paul Bloom
Nature, Color, People 