- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
Tesla drivers run Autopilot where it’s not intended — with deadly consequences::undefined
You must log in or # to comment.
This is about as predictable a failure as passwordless logins. If you can’t secure your software product against it being used outside its intended use case, then stop, go back, you fucked up.
To me, how is this different than someone using cruise control on a 1999 car and reading a newspaper while he blows through stop signs and smashes into a wall. Driver error, reset try again.
Difference is that Elon Musk has claimed since 2016 that this technology will be ready next year. And that it drives safer than a human. And Tesla calls it things like fully autonomous driving and autopilot. Which clearly indicates the car can drive itself safely, when it’s not even close.
You’re absolutely right, but I want to add that there are meaningful, practical differences as well.
The reality is that cruise control doesn’t tend to create accidents because by its very nature it still requires an almost constant level of engagement from the driver. There are very few places where you can run a vehicle on cruise with literally zero user input for more than a few minutes without starting to veer off the road. It assists the driver, but it doesn’t replace their role.
FSD does replace the driver, right up until the moment where the driver needs to step in and correct it. Psychologically, this is a very different interaction. Automation blindness kicks in. If we spend 99% of our time trusting the actions of the machine it becomes very, very difficult to maintain enough focus and attentiveness to recognise the 1% of times when we need to override the machine (this happens in all instances of human oversight over automated processes).
An important difference is that cruise control is simpler to understand. It’s a basic mechanic dressed up as a driver aid. A smaller slice of the population will incorrectly use cruise control.
FSD is a driver aid dressed up as… well, “Full Self-Driving.” It’s not Full, and it’s not Self-Driving. It’s mostly functional in limited circumstances and even then requires driver attention.
I think another good example is how people would never allow a Stasi agent to live in their house, unless the Stasi agent was redefined as a slew of websites, a collection of disparate laws, and multiple steps involving technology.
This is talking about Autopilot not FSD.
Autopilot and FSD are not the same thing though. Autopilot is just TACC + lane-keeping, it’s not advertised as a full self-driving feature.
To drive a car, a driver must:
- accelerate
- maintain speed
- slow for changing road conditions
- make turns
- maintain lane position
- change lanes
- brake for obstacles
- be aware of surrounding cars
- read road signs and respond
- navigate roads to reach a destination
Cruise control can:
- accelerate at the driver’s input
- maintain speed
- slow at the driver’s input
Autopilot can:
- accelerate
- maintain speed
- slow for changing road conditions
- make turns
- maintain lane position
- change lanes
- brake for obstacles
- be aware of surrounding cars
- read road signs and respond
- navigate roads to reach a destination
Which one sounds like it drives the car on its own? Which one is clearly misunderstood by the average driver due to a reference to a feature in an industry experienced by very few people by comparison?
Exactly. Or, using the adaptive cruise control and lane keeping that many cars have these days. (Regular) Autopilot is becoming less of a unique feature of Teslas.
I would like to believe (but lack data to point to to support it) that ADAS is making roads safer overall. There are cases that aren’t covered yet, and driver complacency is a problem for those, but so is complacency in a driver’s belief that they can stare at a phone in their lap but not drift out of their lane and cause an accident, which is something ADAS will protect against.
deleted by creator
A Tesla driving on Autopilot crashed through a T intersection at about 70 mph and flung the young couple into the air, killing Benavides Leon and gravely injuring Angulo. In police body-camera footage obtained by The Washington Post, the shaken driver says he was “driving on cruise” and took his eyes off the road when he dropped his phone.
But the 2019 crash reveals a problem deeper than driver inattention. It occurred on a rural road where Tesla’s Autopilot technology was not designed to be used. Dash-cam footage captured by the Tesla and obtained exclusively by The Post shows the car blowing through a stop sign, a blinking light and five yellow signs warning that the road ends and drivers must turn left or right.
It’s insane that people are blaming this on Autopilot when there is a driver sitting behind the wheel who also missed a stop sign, blinking light, and five yellow warning signs while driving at 70MPH. You could physically do this with any other car that has cruise control and nobody would be blaming the car.
I haven’t seen anyone blaming the autopilot for situations like this
Your above comment was blaming it. Why even bring Autopilot up otherwise?
The above comment, which was a summary of the article, doesn’t blame autopilot. It brings up autopilot as being used by an inattentive driver outside autopilot’s intended use conditions. Acting like autopilot, it’s marketing, and it’s general population perception is an innocent bystander in this situation is, however, disingenuous. You don’t give a car to someone and say “it has airbags, it’s safe” and trust that they’ll actually be ok on the road with no further info, right? So why would you think releasing untested software in a product with overhyped marketing using unfamiliar terms^1 would just be ok?
- The gen pop thinks autopilot can land planes. Any autopilot.
I’ll blame the auto pilot. It’s good enough to train people to not pay attention, but not good enough to be fully driverless. So the users are being trained to fully trust something they can’t fully trust.
If you were trying to teach a new driver how to drive, you wouldn’t do 99.9% of the driving for them, and then randomly throw them into the driver’s seat when there’s an emergency. That’s not how you would get a good driver, that’s how you would get a bunch of accidents. We know that at the human level. If you want to train a driver, you let them practice on the easy stuff, you keep them engaged, you keep them thinking about it.
The Tesla semi-automated self-driving, is the reverse, all the easy stuff the computer does, the rare emergency difficult stuff the human has to do. But they get no practice. It’s like the gold standard of how to create accidents
In user manuals, legal documents and communications with federal regulators, Tesla has acknowledged that Autosteer, Autopilot’s key feature, is “intended for use on controlled-access highways” with “a center divider, clear lane markings, and no cross traffic.” Tesla advises drivers that the technology can also falter on roads if there are hills or sharp curves, according to its user manual. Even though the company has the technical ability to limit Autopilot’s availability by geography, it has taken few definitive steps to restrict use of the software.
Tesla told NTSB that design limits for Autopilot would not be appropriate because “the driver determines the acceptable operating environment.”
He said Tesla could easily limit where the technology can be deployed. “The Tesla knows where it is. It has navigation. It knows if it’s on an interstate or an area where the technology wasn’t designed to be used,” he said. “If it wasn’t designed to be used there, then why can you use it there?”
In a sworn deposition last year first detailed by Reuters and obtained by The Post, Tesla’s head of Autopilot, Ashok Elluswamy, said he was unaware of any document describing limitations on where and under what conditions the feature could operate. He said he was aware of some activation conditions for Autopilot, including the presence of lane lines, and that it is safe for “anyone who is using the system appropriately.”
Tesla’s commitment to driver independence and responsibility is different from some competitors, whose driver-assistance technologies are loaded with high-definition maps with rigorous levels of detail that can tip vehicles off to potential roadway hazards and obstructions. Some manufacturers, including Ford and General Motors, also only allow the technology to work on compatible roadways that have been meticulously mapped.
Over the years, NTSB has repeatedly called on NHTSA to rein in Autopilot. It also has urged the company to act, but Homendy said Tesla has been uniquely difficult to deal with when it comes to safety recommendations. Tesla CEO Elon Musk once hung up on former NTSB chair Robert Sumwalt, said the former chief, who retired from the agency in 2021 when Homendy took over.
Maybe that’s what Elon Musk meant, when he claimed AI is dangerous for humanity.
No shit…
IMHO this comes down to how those features are sold and explained to drivers:
-
if Tesla expects drivers to read through the manual and understand how FSD and autopilot work in order to use them safely it’s on Tesla. People simply don’t do it (not for Tesla or any other cars) and you have to make sure your features are intuitive and easy to understand. If they are not you’re doing it wrong
-
if this is explained clearly to people when they buy the car, there are warning messages and clear instructions and people still use them wrong it’s on the drivers. There will always be stupid drivers, you should not disable new features just because 1% of drivers are too stupid to use them.
-
This is the best summary I could come up with:
After a long day of fishing in Key Largo, Fla., Dillon Angulo and Naibel Benavides Leon pulled to the side of the road and hopped out of their Chevy Tahoe to look at the stars.
The crash is one of at least eight fatal or serious wrecks involving Tesla Autopilot on roads where the driver assistance software could not reliably operate, according to a Post analysis of two federal databases, legal records and other public documents.
While NHTSA has several ongoing investigations into the company and specific crashes, critics argue the agency’s approach is too reactive and has allowed a flawed technology to put Tesla drivers — and those around them — at risk.
In a sworn deposition last year first detailed by Reuters and obtained by The Post, Tesla’s head of Autopilot, Ashok Elluswamy, said he was unaware of any document describing limitations on where and under what conditions the feature could operate.
Tesla’s commitment to driver independence and responsibility is different from some competitors, whose driver-assistance technologies are loaded with high-definition maps with rigorous levels of detail that can tip vehicles off to potential roadway hazards and obstructions.
In 2021, NTSB sent another letter to NHTSA about Autopilot, calling on the agency to “include sensible safeguards, protocols, and minimum performance standards to ensure the safety of motorists and other vulnerable road users.”
The original article contains 2,180 words, the summary contains 224 words. Saved 90%. I’m a bot and I’m open source!
