I have the voice-activated hands-free in my car.  

Like you, I have it on ignore while driving.  I do answer it as a passenger.

I survived for over 50 years of driving without talking on the phone at the same time; I still can.