While I understand it and have in the past supported it, I'm not so keen on it now for the very reason you suggest. It marginalizes many voters. If you're in a strongly blue or red state and you're of the opposite party, there's no point in your participation. That leads to apathy among voters.