A detailed discussion of Moore’s Law for autonomous driving

The hype about driverless cars has been going on for several years. Some optimistic companies even claim that they will be commercially available in 2019 or 2020. But recent fatal accidents and Waymo 's commercialization efforts, which have been loud but not effective, have made people wonder whether driverless cars are out of reach.

How long will it take to realize truly driverless cars? The author of this article is Edwin Olson, CEO of driverless car startup May Mobility. In response to this question, he used the exponential development of technology as a hypothesis and tried to discover Moore's Law for driverless cars through an indicator, and then deduced the approximate time to realize driverless cars, which may be much longer than you think.

As the CEO of a self-driving car company, I'm always asked how long it will take for self-driving taxis to take people anywhere, at any time. We've heard estimates from both salespeople ("Company X will have solved the self-driving taxi problem in 2019!") and engineers ("Ugh, that's hard"), so who can be trusted?

For this article, let’s measure a system’s performance in miles per disengagement (MPD). A disengagement is roughly defined as when the technology fails and the safety driver needs to take over. For a good self-driving car, this number will be high — meaning the car can drive long distances on its own and rarely fails.

Moore's Law?

To answer the question at the beginning, I decided to use Moore's Law as a comparison. Moore's Law is an empirical observation that the number of transistors doubles every 18 months. This is exponential growth - it's Moore's Law that made your phone faster than your computer in 2000.

Exponential growth is rare. For example, trees and people grow linearly, which is much slower. Most things that grow exponentially don't survive; for example, bacteria reproduce exponentially before they get too crowded. In fact, Moore's Law seems to have broken down for computers!

That being said, it’s not unusual for technology to advance at an exponential rate in its early days. This is an optimistic assumption, but if you want to make bold futuristic predictions about how quickly the world will change because of technology, you should assume exponential growth.

So now, let's make a bold and optimistic prediction about the future of self-driving cars. We're going to assume that there will be exponential growth in technological improvements. In other words, we're going to calculate Moore's Law for self-driving cars. But you're probably not going to like the answer.

data

In 2004, the best self-driving car was CMU's Sandstorm, which "won" the first DARPA Grand Prix by completing 7.4 miles of a 150-mile race before getting stuck on an embankment, its tires spinning in vain until they emitted white smoke (not a knock on it, other cars did worse!). Let's discount its failure rate and say it failed once every 10 miles.

In 2018, Waymo's data was 11,017 miles per disengagement (defined by California as "technical failure"). This is roughly equivalent to 10 to the fourth power miles per disengagement.

With these two data points, we can calculate Moore's Law for self-driving cars.

…the number of miles disengaged from autonomous driving is doubling approximately every 16 months…

Moore's Law for self-driving cars is almost identical to Moore's Law for computers - performance doubles every 16 months! This is a cosmic coincidence!

The black line above represents the progress of self-driving cars from the DARPA Urban Challenge in 2004 to 2018. We can extrapolate that line (red) and see where it crosses human performance (blue). The Y-axis is a logarithmic scale, so exponential growth will be reflected in a straight line.

The key question is “How good does the system need to be?” Let’s assume the goal is to match human performance. Humans are actually very good drivers, with only one fatal accident every 100 million (10^8) miles! To put this in context, the average human driver drives a few hundred thousand miles in their lifetime. Each self-driving car probably has less than 20 million miles driven.

So far, the gap between human performance (10^8 miles between fatal accidents) and the best self-driving cars (10^4 miles per hands-off) is 10,000 times better. In other words, self-driving cars are 0.01% as good as humans.

Even if performance doubles every 16 months, it will take 16 years for self-driving cars to catch up with humans - that's 2035. Claims that self-driving cars will be available by 2019 or 2020 are beginning to look suspicious. (Of course, we see eye-catching demonstrations by self-driving car companies, but that's just to show off their technology. It doesn't necessarily mean their systems perform as well as humans!)

Many driverless car failures only result in injuries rather than fatalities. The distance between two human injuries is "only" 10^7 miles, so if we assume that driverless car failures never result in fatal accidents (just injuries), the previous prediction can be reduced by 4 years. But it will still take 12 years to reach human performance.

So to sum up, there are a few points to note:

The performance of driverless cars doubles approximately every 16 months. This is Moore's Law for driverless cars.

Currently, the performance of driverless cars is roughly equivalent to 0.01% of human drivers, and self-driving taxis may still be a dream before 2035.

There are two holes in this prediction. New technologies could emerge that change this growth curve. Or the company could decide to pursue an application that is slightly less difficult than "anywhere, anytime".

This may be bad news for self-driving taxi companies, but good news for commuter companies.

appendix

If you're detail-oriented, here are 3 more things to consider:

The first point

The "Moore's Law" for driverless cars depends on the data we use. If you think that driverless car companies are overly optimistic about the performance of their cars, the realization of self-driving taxis may have to be pushed back.

On the other hand, if you believe that today’s best self-driving cars drive 10 times better than Waymo’s public numbers (e.g. 110,000 hands-free miles per trip), then the rate of improvement is much faster than the above calculation. But even if today’s systems drive that well, it will still take until 2028 to reach human driver performance.

Second point

A key assumption made earlier in this article is that technology improves at an exponential rate. This assumption implies that the disengagement rate is doubling every 16 months. This number is optimistic: we cherry-picked a very low automation rate in 2004 and cherry-picked the best commercial data from 2018, which tends to create a picture of rapid progress.

To see how optimistic this assumption is, look at public reporting in California. Are AV companies growing at an exponential rate? If so, is that rate faster or slower than 16 months?

Waymo's data is provided on an annual basis, and I could only find 4 years of data. Note that this curve is not logarithmic like the one above; if the trend was exponential, we would see a curve that bends sharply upward. Waymo's 2018 report performance was twice that of 2017, but 2017 was basically the same as 2016 (and 2016 was a good year relative to 2015!). What does this mean? We can use an exponential fit to this data and conclude that it doubles every 16 months. (Note: There are many ways to fit the data. I set 2015 as the first year and then used the least squares method to fit it in the form of A*exp(Bt)) But this fit is actually quite poor - if you want to know, see Appendix 2. For the sake of discussion, these 4 data points look equally credible with a linear fit (in which case self-driving taxis may not be realized until 20,000 years later.). But isn't it elegant that it appears again every 16 months?

Cruise's monthly data is also moving upward, but the data is very noisy. Using the same fitting strategy above, we will get a doubling of performance every 18 months.

Although real-world data is full of challenges, it is interesting that the final conclusion is close to 16 months. This gives some credibility to the idea that everyone is developing at the same level, and therefore self-driving taxis will probably not be realized before 2035. Unless, the third point.

Third Point

California’s definition of disengagement excludes many types of interventions, so it is inherently an optimistic measure of the maturity of the technology. In other words, these disengagement figures are only an approximation of situations where companies expected the technology to work but experienced system failures, and do not capture situations where the technology was not expected to work.

So of course the system performs badly in situations where you don't expect it to work! But if the question is "how close are we to viable self-driving taxis that can go almost anywhere?", there's a whole class of driving scenarios that aren't considered that should give you pause.

Waymo’s 2016 disclosure sums it up well:

The DMV rule defines a disengagement as the deactivation of the autonomous mode in two situations: (1) “when a failure of the autonomous technology is detected,” or (2) “when safe operation of the vehicle requires that the driver of the self-driving vehicle disengage the autonomous mode and immediately assume manual control of the vehicle.” Based on this definition, the DMV noted, “This clarification is necessary to ensure that manufacturers do not report every common or routine disengagement.”

As part of our testing, our cars switch into and out of autonomous mode many times a day. These disengagements can number in the thousands each year, though the vast majority can be considered routine rather than safety-related. Safety is our highest priority, and Waymo test drivers are trained to take manual control in many situations, not just when safe maneuvers “require” them to do so. Our drivers err on the side of caution and will take manual control if they have concerns about the safety of continuing in autonomous mode (for example, due to the behavior of a nearby self-driving car or any vehicles, pedestrians, or cyclists), or when other concerns warrant manual control, such as improving ride comfort or easing traffic flow. Similarly, the car’s computer will hand control back to the driver in many situations that do not involve a “failure of the autonomous technology” and do not require the driver to take over immediately…

Appendix 2

Below are the curves fitted to the Waymo and Cruise data. These models fit pretty well for the most part, but I think you can safely conclude that it's pretty hard to justify a much higher rate of improvement than what's calculated here.

Waymo’s MPD overlaid with an exponential fit. The exponential curve corresponds to a doubling of performance every 16 months, but this fit is actually pretty bad.

Cruise's MPD is superimposed with an exponential fit. This fit is particularly questionable because there is some noise. But it's not clear whether the exponential fit is appropriate.

Here is Cruise data again, with the peak that caused the problem removed manually. This type of data tinkering is statistically inappropriate, so take it with a grain of salt. This curve fit shows that performance can double every 7.5 months, but the curve doesn’t actually fit the data, especially the right side fit is already higher than the actual data.