Self-driving vehicles.

[jderimig]

I've been eagerly wainting to see this statistic:

"According to this report, the average Tesla equipped with FSD Beta, driven on predominantly non-highway sections of road, crashes 0.31 times per million miles, a dramatic decrease from the average American, who crashes 1.53 times every million miles."

https://www.teslarati.com/tesla-fsd-safety-statistics/
First clear demonstration I know of that a robot can drive more safely than a human.

Aside from this being Tesla's report it demonstrates the a robot can drive more safely than some humans. The "average" American in my observation is a terrible driver and 50% are worse than that.

This human has a crash rate of 0 per million miles.

 

[jderimig]

I would also like to see the raw data from this number. Questions I would have are:
1. How many crashes did they have? Then we can actually calculated on confidence interval to actually compare to the 1.53 average driver benchmark.
2. To get 0.53 there must have been some crashes. What kind of crashes were they? Were they slight scraping of a guard rail or were they running into a stopped school bus? Stuff like that matters.
3. Were there actually crashes or were there interventions? Or both?

 

[boatgeek]

I would also like to see the raw data from this number. Questions I would have are:
1. How many crashes did they have? Then we can actually calculated on confidence interval to actually compare to the 1.53 average driver benchmark.
2. To get 0.53 there must have been some crashes. What kind of crashes were they? Were they slight scraping of a guard rail or were they running into a stopped school bus? Stuff like that matters.
3. Were there actually crashes or were there interventions? Or both?

1. The Teslarati article says that NHTSA is investigating 40 collisions. So >40, not clear how many. The source for the article is here, and you can see a dot cloud of impact locations on page 38. That gives a sense of how many crashes (guess >200, but still hard to read).

Call me a skeptical bastard, but I also get somewhat suspicious when the company report breathlessly states some of the findings but clearly leaves others out. The linked article gives these data points for Full Self Driving, Autopilot (driver assistance), and neither:
On predominantly non-highway sections of road, FSD has 0.31 crashes/M miles, average American has 1.53/M miles.
On highways, Autopilot has 0.18 crashes/M miles, and Tesla drivers using neither Autopilot nor FSD have 0.68 crashes/ M miles.

This all sounds great, but where are the stats for FSD on highway? What about Autopilot or unassisted Tesla drivers on non-highways? Tesla obviously has those numbers, but they don't appear to be releasing them. Beyond that, if we were going to do a real statistical analysis, we'd want breakdowns of demographics (are relatively wealthy people less likely to get into car accidents?) and state-level accidents (there's substantial differences in traffic fatality rates between states).

This site has an interesting take:

Well, in short, it's inconclusive, but there is a justified need for caution. There are variables that could more than explain away any performance increases shown in the data that are not related to the cars themselves. There is potentially even an argument that the cumulative nature of the explanatory figures shows that the cars on active autopilot are actually worse, and we are also concerned that Tesla have removed the one related statistic which would have been useful.


That said, it's good to see the willingness to release these figures, even if we suspect Tesla are being somewhat economical with the story telling. Unfortunately, the anti-Tesla people can find much to dispute the figures, whereas the pro Tesla lobby will take them at face value and claim they are evidence of Tesla's progress. We're somewhere between the two, the figures overstate the reality, but they're still good to see, we just wish there was more transparency and more of a scholarly presentation of the data that satisfied those, like ourselves, with a genuinely curious mind.

 

[jderimig]

1. The Teslarati article says that NHTSA is investigating 40 collisions. So >40, not clear how many. The source for the article is here, and you can see a dot cloud of impact locations on page 38. That gives a sense of how many crashes (guess >200, but still hard to read).

It is not clear if those 40 are in Telsa's dataset. It mentioned a specific firmware version over non-highway roads. So I am not sure what the environment was. Comparing their data reduction (average 0.53) to average driver (whatever that is) under unknown driving environment comparison is not really technical result but a marketing puff piece.

 

[Funkworks]

I would also like to see the raw data from this number. Questions I would have are: …

It is not clear if those 40 are in Telsa's dataset. It mentioned a specific firmware version over non-highway roads. So I am not sure what the environment was. Comparing their data reduction (average 0.53) to average driver (whatever that is) under unknown driving environment comparison is not really technical result but a marketing puff piece.

The report is certainly finalized by a marketing team (like the specs of any other product), but the numbers are credible (like the specs of any other product). I see no reason to think they would be way off (like the specs of any other product). I’m not sure if the NHTSA or anyone else will publish a more detailed report with similar results about Tesla’s FSD to compare with. This the best I have so far.