Open borders are scary probably because the upside is relatively limited for rich countries, because we are already rich. We can either make ourselves a little richer, or on the downside we can make ourselves <i>a lot</i> poorer, by becoming overcrowded with worldwide economic refugees.