It is indeed troubling that Clinton could be driven out by voters who don't necessarily deplore his conduct.
The voters don't necessarily oppose Clinton's conduct.
Do you not comprehend the meaning of my thoughts?