Even when it comes to ethical problems, people forget one thing: They don't HAVE to be perfect. they just have to be better than us. Spoiler Alert, they already are.
They have to be better - and then some. An ideal future would be one where automated vehicles do most of the transportation for us in such a way that it is just as safe as, say, air travel. But to get there, you can't just present a safe AV and be done. They need to gain people's trust. They need to drive courteously, reliably. They shouldn't randomly try to kill their owners. I don't think just being safe enough will cut it. Safety, especially traffic safety, is an abstract and distant concept until it's not and you're in the hospital. I recently drove in one of the first self-driving buses in existence, and it was a bit of a shaky ride as the car had a hard time accelerating and braking smoothly. While I knew that bus, with its own bus lane, has never been in a serious accident in 10 years, my gut feeling was that it wasn't much safer than a regular bus.