Waymo simulation study shows hypothetical big reduction in fatal crashes

[ImaCynic]

Wonder how many environmental variables were considered in the simulation… On the drunk driver example, what if there was another oncoming car behind the drunk driver? How would Waymo choose which head-on collision is “better”?

You may recall the Uber autonomous car that struck and killed a woman crossing the road with her bike few years back? Here’s a good article on the likely causes:

https://www.theinformation.com/articles/uber-finds-deadly-accident-likely-caused-by-software-set-to-ignore-objects-on-road

I suspect much has improved since this incident, but still, the machine is only as good as the hooman that programs it.

[trailrunner]

I’ve spent some of my career in autonomous vehicles and simulation.

I’m extremely skeptical of their claims.

[peterw_diy]

My IT perspective: sounds like bullshit that I’d expect from a PR person misrepresenting what you get with some modern-ish software development techniques like test-driven development and machine learning.

You do your best to build a computer system that handles whatever input comes its way. And to test your work, you throw Test Cases at it, and make sure you get the Desired Results. You adjust your system until you get the Desired Results. And then you pray that the systen can also cope with new inputs after it Goes Into Production.

If you’re building an autonomous car, you collect data about past crashes and use that as some of your Test Cases. You make sure your simulation of those past tragedies results in your simulated auto avoiding the same simulated awful outcome. Real driver killed a Crazy Ivan? Use the simulator to try to ensure your auto executes an acceptable course of action in a simulated copy of that scenario.

But in real life, you’re gonna get a different Crazy Ivan, in a different situation. Maybe this new one is wearing grey and harder to see. Maybe his speed is less conssitent. Maybe there are two Ivans. Or three. Or it’s an irregular intersection. Or there’s significantly different weather. The fact that your simulation (which is likely incomplete — are you really accounting for all factors? Debris on roadway? Asphalt condition? Impefect wheel balancing? Glare off oncoming windshields?) avoided the one well-documented Ivan from seven years ago doesn’t mean you’re gonna not crash into next week’s Ivan. Especially if you pulled a Volkswagen emissions hack and tailored your system to the data about those past incidents.

[dkel]

@peterw_diy 210414 wrote:

My IT perspective: sounds like bullshit that I’d expect from a PR person misrepresenting what you get with some modern-ish software development techniques like test-driven development and machine learning.

You do your best to build a computer system that handles whatever input comes its way. And to test your work, you throw Test Cases at it, and make sure you get the Desired Results. You adjust your system until you get the Desired Results. And then you pray that the systen can also cope with new inputs after it Goes Into Production.

If you’re building an autonomous car, you collect data about past crashes and use that as some of your Test Cases. You make sure your simulation of those past tragedies results in your simulated auto avoiding the same simulated awful outcome. Real driver killed a Crazy Ivan? Use the simulator to try to ensure your auto executes an acceptable course of action in a simulated copy of that scenario.

But in real life, you’re gonna get a different Crazy Ivan, in a different situation. Maybe this new one is wearing grey and harder to see. Maybe his speed is less conssitent. Maybe there are two Ivans. Or three. Or it’s an irregular intersection. Or there’s significantly different weather. The fact that your simulation (which is likely incomplete — are you really accounting for all factors? Debris on roadway? Asphalt condition? Impefect wheel balancing? Glare off oncoming windshields?) avoided the one well-documented Ivan from seven years ago doesn’t mean you’re gonna not crash into next week’s Ivan. Especially if you pulled a Volkswagen emissions hack and tailored your system to the data about those past incidents.

My parenting perspective: this sounds pretty much exactly like me sending either of my teen sons out in the car after teaching them to drive. Everyone operates from their own experience, and after that, everyone makes it up as they encounter things outside their experience, hoping for a good outcome. (This is not an endorsement of AI drivers; just noting that we learn and function in the same way, in many respects, if we’re really honest.)

[Steve O]

I skimmed through the actual study. In all of the cases in which the original driver was drunk or drowsy or stupid, the autonomous car was not. Those crashes didn’t happen, which I believe is a pretty fair assumption.
I am 100% confident that autonomous cars will be involved in far fewer crashes than human drivers. I am also 100% sure they will be in some.
The authors do not claim that autonomous cars will not crash, only that when they recreated this particular set of fatal crashes, no one died in the simulations.

Sent from my H3123 using Tapatalk

[trailrunner]

Quote:

“The simulated Waymo Driver completely avoided or mitigated 100% of crashes, aside from the crashes in which it was struck from behind, including every instance (20) that involved a pedestrian or cyclist,”

@Steve O 210417 wrote:

The authors do not claim that autonomous cars will not crash, only that when they recreated this particular set of fatal crashes, no one died in the simulations.

The bumper sticker you posted in your first post implies something different — that there is at least some significant decrease in accidents altogether and across the board. “Completely avoided”! Yes, this is the click-bait internet we live in, and details, nuances, and caveats matter, but that quote would lead the reader to believe that autonomous vehicles are the way to automotive safety.

If their only standard is to decrease fatalities, I can do that by driving 5 mph. Or completely stopping whenever I detect something, but can’t classify or identify it.

We are a long way from unleashing autonomous vehicles on our roads.

[mstone]

@trailrunner 210418 wrote:

If their only standard is to decrease fatalities, I can do that by driving 5 mph. Or completely stopping whenever I detect something, but can’t classify or identify it.

If we could just convince human drivers to just f’ing slow down or stop when they’re confused we’d probably have 10k fewer deaths every year. But no, human drivers just keep truckin’.

[Steve O]

@ImaCynic 210412 wrote:

Wonder how many environmental variables were considered in the simulation… On the drunk driver example, what if there was another oncoming car behind the drunk driver? How would Waymo choose which head-on collision is “better”?

Your wondering can be ameliorated by reading the actual study.

[Steve O]

@trailrunner 210413 wrote:

I’ve spent some of my career in autonomous vehicles and simulation.

I’m extremely skeptical of their claims.

Their claim is that when they simulated this set of fatal crashes in Arizona using a specific methodology, which is described in the study, none of them resulted in a fatality in the simulations.
Which part of that are you skeptical of?

[Steve O]

@trailrunner 210418 wrote:

The bumper sticker you posted in your first post implies something different — that there is at least some significant decrease in accidents altogether and across the board. “Completely avoided”! Yes, this is the click-bait internet we live in, and details, nuances, and caveats matter, but that quote would lead the reader to believe that autonomous vehicles are the way to automotive safety.

My bumper sticker is accurate: The simulations “completely avoided or mitigated” 100% of the crashes. I do not take responsibility for how the journalist wrote the article, but I did read the study itself (well, skimmed), and the study, IMO, is fair in how it characterizes what they studied, how they carried it out, and their results.

Depending on what you mean by “automotive safety,” I do believe that widespread adoption of autonomous vehicles will result in “at least some significant decrease in accidents altogether and across the board,” yes. Particularly as a bike rider, this makes me feel good.

If by “automotive safety” you mean complete elimination of all crashes, then I agree that is likely impossible.

[trailrunner]

@Steve O 210423 wrote:

Which part of that are you skeptical of?

Quote:

Waymo says autonomous cars could have prevented most fatal Chandler crashes in 10-year period

…..

[trailrunner]

@Steve O 210424 wrote:

My bumper sticker is accurate: The simulations “completely avoided or mitigated” 100% of the crashes. I do not take responsibility for how the journalist wrote the article, but I did read the study itself (well, skimmed), and the study, IMO, is fair in how it characterizes what they studied, how they carried it out, and their results.

Sorry, but I disagree. The bumper sticker (and headline) clearly implies that autonomous vehicles can eliminate all accidents. Apply the reasonable person standard.

If I had put that bumpersticker on one of my briefings, I’d be in big trouble (understatement) because it’s misleading and doesn’t tell the entire study. (OTOH, I’m an engineer, not a marketer.)

Companies have been overselling autonomy for a long time. It’s not there yet. I’m listening to a company presentation right now. It’s not there yet.

@Steve O 210424 wrote:

Depending on what you mean by “automotive safety,” I do believe that widespread adoption of autonomous vehicles will result in “at least some significant decrease in accidents altogether and across the board,” yes. Particularly as a bike rider, this makes me feel good.

Autonomous vehicles terrify me, and not because I fear skynet. I have developed hardware and software for autonomous vehicles and I know how hard it is, and I know how what the current state of the art is.

I agree that distracted driving, impaired driving, road rage, and speeding are major human factors leading to car crashes. Autonomous vehicles presumably wouldn’t have these problems, so they have that potential advantage. Someday we may get there, but the timeline is not measured decades, not years.

[Steve O]

@trailrunner 210427 wrote:

Sorry, but I disagree. The bumper sticker (and headline) clearly implies that autonomous vehicles can eliminate all accidents. Apply the reasonable person standard.

Okay. I took the bumper sticker off my original post. That sentence, of course, is still in the article, but is surrounded by more context. The study is available for anyone to read for themselves.

[Steve O]

@trailrunner 210427 wrote:

Autonomous vehicles terrify me, and not because I fear skynet. I have developed hardware and software for autonomous vehicles and I know how hard it is, and I know how what the current state of the art is.

I agree that distracted driving, impaired driving, road rage, and speeding are major human factors leading to car crashes. Autonomous vehicles presumably wouldn’t have these problems, so they have that potential advantage. Someday we may get there, but the timeline is measured in decades, not years.

These things you point out in your second paragraph terrify me even more.

I will take 100% autonomous cars in their best current state of development at the intersection of Lynn and Lee in Arlington above human drivers. Today.

Like most technologies, it will not happen all at once. Autonomous vehicles are already being used in certain contexts (campuses, for instance). They will expand their presence as they get better. I will bet you a month’s salary that as autonomous vehicles as a share of vehicles on the road grows and the share of human drivers declines, fatalities and injuries to all road users will also decline. I, for one, can hardly wait.

[mstone]

@trailrunner 210427 wrote:

I agree that distracted driving, impaired driving, road rage, and speeding are major human factors leading to car crashes. Autonomous vehicles presumably wouldn’t have these problems, so they have that potential advantage.

Unless someone programs in an ego, or tendency to start thinking about things other than driving, or a fear that driving defensively will make their reproductive organs smaller, autonomous vehicles have many advantages. Simply not speeding through residential neighborhoods would cut the death rate, even if they hit the same number of people.

[Author]

Posts