Under the overmechanical assumptions of affirmative-action opponents themselves (and putting aside the racial IQ theories of Murray and some others), blacks would move up the list, and whites would move down.
Affirmative-action would move whites up the list and blacks down the list.