In the past , the most obstinate white opponents of integration ( the Goldwater wing of the Republican Party and conservative Southern Democrats ) argued that people should have the right to associate--or not to associate--with whomever they wish .
White opponents of integration supported their argument on the basis of freedom of association or non-association more specifically .