I see what you mean now. This plus the comment from user `argonaut` helped me realize what I was missing. Not all of generalization error is because of overfitting, and even if none of it is because of overfitting, the idea you describe would <i>still</i> allow for the attack.