For both genders, substantial portions of the U.S.-born population of working age are not in the labor force because they are attending school, retired or disabled, but that is true for only a small share of unauthorized immigrants.