In the days since a self-driving Uber vehicle killed a pedestrian in Tempe, Ariz., local police and federal agencies have yet determine whether Uber’s technology was at fault.
But new documents obtained by The New York Times show Uber’s technology had made little progress in the last year. The story details the series of setbacks the company faced in trying to get self-driving cars to market, including careless safety drivers who are supposed to take over test cars in case of emergencies.
The key stat underpinning the report is what’s known as “miles per intervention,” or the number of miles the car can drive on its own before the safety driver has to take over the car. The Times story cites internal documents showing Uber was unable to meet its goal of driving an average of 13 miles without a driver having to take back control as of March.
That’s not much better than its rate of intervention this time last year. In March 2017, documents Recode obtained showed Uber’s safety operators had to take back control of the cars an average of once per every 0.8 miles.
Uber wanted to spell out what this metric means and doesn’t mean in the wake of the fatality and sent this statement:
MPI is not a measure of the overall safety of our testing operations, and shouldn’t be interpreted as such. Miles per intervention is one of many metrics that we use to track our system’s improvement, but without context it can be one of the least useful. For example, depending on where and how it’s tested, the same software could result in significantly different MPI. Additionally, companies may define interventions differently from each other.
In other words, miles per intervention is a broad metric that includes most of the times drivers have had to take back control from the system over the course of a week.
The reasons for these interventions can include navigating unclear lane markings, the system overshooting a turn or driving in inclement weather. The stat excludes accidental disengagements, end-of-route disengagements and early takeovers.
Other metrics include the average number of miles between “critical” interventions — when a driver has to avoid causing harm, such as hitting pedestrians or causing material property damage and the average number of autonomous miles between “bad experiences” — things like jerky motions or hard braking, which are more likely to cause discomfort than damage.
Then there’s the total number of miles driven autonomously. The documents the Times obtained indicated Uber had driven 3 million miles as of March.
While other companies may define interventions differently, Uber’s rate of intervention in Arizona is far more frequent than that of its competitors’. For example, Alphabet’s self-driving company, Waymo, had a rate of 5,600 miles per intervention in California.
Still, this is by no means an indication that the technology was at fault in the fatal crash. It is, however, an indication of Uber’s slow technological progress with its self-driving cars.