We are not there yet, but in the end I think we will find that is by far safer than drivers or humans being able to take over at some point. The reality is the autonomous cars can and will react in a microfraction of the time we take to respond. They will avoid rear end accidents better and faster, and will indeed reduce them by a ton (almost all are caused by folks traveling too closely and/or losing attention, where those will calculate speed and distance and likely other factors to A not be in a situation where they are just way too close and B will react so much faster then we ever do once a situation arises). I think adding humans to the mix make it a lot worse than it's worth, but again not for quite a long time. Like I said, I think relying on humans to take over is the worst of the long term options because humans who are not in total control won't pay near as much attention, and we don't pay enough even when we are in control.
So we'll only be able to go where we're allowed to go by the limits of the navigation system and public infrastructure. Every stop planned in advance and tracked. People that have no idea how to drive if they ever leave the most-developed countries. No thanks.
I hope that by the time this happens I am six feet under or out in the sticks way way off the grid.