It's a temporary issue.  Eventually people will accept them like they accept 'self driving elevators'.  I'm a lot more worried about some drunk crossing the center line and killing me, or some twit running a red light while texting.  Once people start to experience the convenience and safety of a self-driving car, they'll be broadly accepted.