Phil & Junko on AV Safety

When a human-driven vehicle gets into a crash, an established legal framework holds parties who have contributed to harm accountable. But how does that work if a misbehaving robotaxi or robotruck causes harm?

“We really don’t know yet” is the rather unsettling answer from Michael Brooks, executive director of The Center for Auto Safety. Brooks is the guest on our latest video podcast on AV accountability.

Our podcast breaks down -- in plain English -- the legal and regulatory issues that must be resolved in order to do right by victims of AI-driven robotaxis.

Ordinary consumers might simply assume that a robotaxi company is responsible if its vehicle is at fault.

Not so fast.

Accountability for autonomous vehicles right now is primarily limited to product liability. But Brooks asks, how can we prove the product had a defect when the fully autonomous vehicle was running on an End-to-End (E2E) AI algorithm? When the AI processing inside a vehicle is done in an inscrutable black box, how do the plaintiff’s lawyers and expert witnesses in a product liability case find source-code defects … if “source code” is even the relevant term for machine learning?

Worse, my podcast co-host Phil Koopman asked, what if the manufacturer insists that the vehicle’s AI is the best one on the market and it would be uneconomic to do better? The manufacturer could even argue that “once in a while, it kills people, but that’s just the way it is.”

Counter-intuitively, the risk-utility test for product defects might mean that occasional dangerous AI behavior does not equal a product defect. Victims might be forced to fund an expensive court case based on novel legal theories to recover compensation for being hit by a robotaxi -- if they aren’t precluded from access to the court system by forced arbitration. Absent other regulatory or legal accountability mechanisms, how would that work out in practice?

This podcast also touches on the AV industry’s biggest complaint: a patchwork of state-by-state AV regulations. Brooks explains the AV industry’s motives behind its effort to preempt state efforts at regulation, at a time when no federal AV regulations specific to ensuring AV safety exist yet.

Lastly, the issue of AV accountability isn’t just about who should be responsible for avoiding harmful outcomes. More importantly, the question is how to incentivize AVs to behave better so that they won’t perpetuate blunders such as robotaxis driving past stopped school buses.

Listen to this episode on Spotify and iTunes Podcasts. Clickable topic links below point to the YouTube version:

2:08 Introduction of Michael Brooks
3:59 Legal framework for AV?
6:17 Comparative, modified comparative, or contributory negligence...
8:40 Why a patchwork of state regulations is a big deal for AV industry?
12:53 What aspect of AV accountability are you concerned about most?
13:01 The black box of AI
14:15 Assume I were hired as an expert witness
16:56 Why I am worried about product liability
17:34 Two paths to liability
19:45 Who should be held accountable when AV causes harm?
23:11 How to incentivize a ‘Computer Driver’
28:17 Driver License point for AVs?
31:12 AV accountability needs a stack of mechanisms
38:38 AV industry wants AVs to be treated differently from humans
40:20 Challenge round 1: What is the one thing you would put into a US Congressional Bill?
40:49 Forced arbitration
45:14 Challenge Round 2: The most promising thing you saw last year, that might improve AV safety?

This week we were delighted to discuss how AVs affect the cities in which they deploy with Rachel Castignoli and David Zipper.

Meet Rachel:

Is the city where you live new to robotaxis? If so, tune in right here to hear Rachel Castignoli, senior business process consultant in Austin, Texas. She’ll recount what the city has learned while adapting its infrastructure, and what she wished she had known before robotaxis came to town.

Castignoli’s team is best known as a pioneer in creating and maintaining a public dashboard of AV incidents in Austin. https://austin.maps.arcgis.com/apps/dashboards/dbd59d9b20634de1afad32a8267b739b#

She comes away with practical advice on how best a city can integrate robotaxis into its transportation system.

Meet David:

For anyone who doesn’t know how self-driving technology got started or about the problems AVs were supposed to solve, this episode will be an eye-opener. The AV industry’s “saving lives” narrative had not emerged until Google got involved in the self-driving car technology. Safety or “saving lives” is a “marketing pitch,” not a mission statement, according to journalist and podcaster David Zipper, who focuses on issues of mobility, cities and technology.

Have AVs reduced congestion in cities? Have they made urban traffic more orderly? Do robotaxis really offer wheelchair access? Do they go to places where they are most needed? Zipper breaks it all down.

The Discussion:

The real zinger on these questions comes from Phil Koopman, safety expert and professor emeritus at Carnegie Mellon University. Koopman defines solving safety problems as table stakes. But the AV industry’s maniacal focus on statistical crash rates has become a distraction from “the real question we ought to be asking.”

That question: “What’s the social benefit of autonomous vehicles?” Put even more simply: “What good are they?”

Zipper challenges the AV industry by asking, “What if we have not just 8,000 autonomous vehicles operating on the streets in America, but 80 million. What is the world going to look like? Is it better, or worse?”

Answering his question with uncertainty, Zipper said, “I don’t think it’s a clear yes” that it will be better.

He continues, “If those who work at AV companies find themselves thinking, ‘Yeah, it’s not totally clear,’ then maybe it’s time to re-think the future of the products you are developing.”

Listen to this episode on Spotify and iTunes Podcasts. Clickable topic links below point to the YouTube version:

4:16 Rachel Castignoli introduction

5:16 David Zipper introduction

7:32 Every AV company has a different vibe

10:10 AV Dashboard in Austin, Texas https://austin.maps.arcgis.com/apps/dashboards/dbd59d9b20634de1afad32a8267b739b#

13:37 AV companies know all the AV incidents

18:34 AV promises made, promises kept?

19:49 The origin of self-driving technology

21:30 AV Hype One

25:48 Do AVs reduce congestion?

29:58 Regulatory bargains are needed

36:2 Are cities informed of software updates?

40:17 Software updates can make things better or worse

44:22 What should be tracked on a “national” AV dashboard

49:04 We need datasets beyond crashes

55:07 AV overturns the justice system

56:50 Tying AV performance with their ability to scale

58:20 Challenge Round 1: Advice to cities new to robotaxis

59:20 Challenge Round 2: If robotaxis are the answer, what was the question?

Human drivers have strengths that are tough for computers to match.

As the press heaps praise on autonomous vehicles’ “computer driver,” for all its so-called “super-human” capability, it might be instructive to probe the good, the bad and the ugly of “human driving.”

Lisa Dorn, Associate Prof. of driver behavior at Cranfield University (Bedfordshire, England) is our podcast guest.

Human drivers are often portrayed as villains who get easily distracted, fatigued or impaired. But surely, a human at the wheel must have some redeeming qualities.

We ask Prof. Dorn about where human drivers genuinely excel, compared to their computer rivals. These redeeming qualities include the human ability to read the road surface, adapt to changing conditions and quickly recover from driving errors.

Also often overlooked is defensive driving, the human knack for making allowances for the mistakes of other road users. Can a computer driver replicate these intuitive, affective and human skills? Or is this too much to ask?

4:34 Background of Lisa Dorn

6:26 The premise of this episode: The focus on human drivers

7:18 Road safety in the UK

9:06 Where human drivers excel

11:56 Allowances for the mistakes made by other road users

12:45 Missing ability of computer drivers

13:55 Stereotype of a computer driver

14:53 Examples of computer drivers’ odd behavior

19:21 Three stages of situational awareness

20:20 Robotaxis and traffic enforcement

22:09 Human’s ability to ‘transfer’

22:54 Give AVs a driver test

25:08 Humans learn by proxy

26:01 Assumptions that shouldn’t be made about computer drivers

28:58 How to program computer drivers so that we can get along

31:54 A pile of robotaxis, if empty, evokes different human emotion?

38:01 What our research in driver performance has taught us

39:28 What Waymo’s ‘Responder Role’ entails

42:11 ‘Traffic Psychology’ is a big missing piece of the jigsaw

47:15 Challenge Round 1 (Should human drivers be banned?)

49:46 Challenge Round 2 (Human + Computer paired creates a safer outcome?)

In this episode, we talk with Nick Reed, founder of Reed Mobility.

In the UK, the Automated Vehicles Act, passed into the law in 2024, expects Autonomous Vehicles (AVs) to behave carefully and competently.

But what does “Careful and Competent” driving entail? And where does the phrase come from?

In the United States, on one hand, we hear lots of statistical safety claims by AV companies. On the other hand, we know very little of the public’s perception and expectation of AV safety.

Instructive is the results of the deliberative survey research conducted by the U.K. government. Nick assisted the research efforts. The survey, involving six cities—both big cities and rural areas, zoomed in on the public perception on careful and competent driving, and how they see it to be applied to autonomous vehicles.

As Nick unpacks in this podcast, we will learn that the public and AV companies use a very different lens from ordinary human drivers when they discuss AV safety.

This episode is a follow-on to Phil’s written piece on Careful & Competent Computer Drivers.

You can listen to this episode on Spotify and iTunes Podcasts. Clickable topic links below point to the YouTube version:

02:14 Nick Reed intro

04:37 The origin of the phrase: Careful and Competent Driver

06:03 The UK Police defines “driving offenses” (See this web site)

07:26 Equivalent to “Careful and Competent Driver” in the US?

11:41 The U.K. government research project on the public’s perception of “Careful and Competent”

12:48 How the public discussed Careful and Competent Driver

13:22 Two different views on Careful and Competent Driver

14:38 It’s the law in the UK: AVs must behave carefully and competently

16:51 Different AV concerns were raised by city dwellers and those in the rural areas in the U.K.

21:00 No “free pass” for AVs

22:28 For AV safety assurance, what evidence would be persuasive?

24:03 What safety evidence can AV companies provide to regulators?

28:21 Vehicles’ behavior vs. Statistical benefits

28:41 Why smart motorways did not take off

30:35 There is more to safety than numbers

33:06 Data sharing is the key

36:57 How do you respond to the argument against “Careful and Competent”?

37:30 Risk meter

39:20 Use the UK police enforcement guidelines on “Careful and Competent”

40:37 Challenge round 1: What about human drivers getting away with bad driving?

43:39 Challenge round 2: If AV companies just did Careful & Competent, would that be enough?

45:52 MHTSA vs. States: Two different regulatory roles

47:39 Persuasion vs. Calibration

52:58 Digital Commentary Driving

This week Phil & Junko talk about what an edge case is, and why autonomous vehicles will need the help of remote human assistants to resolve edge case issues.As robotaxi operations have begun to scale, the public is seeing more incidents in which autonomous vehicles caught in edge cases struggle to get out of the bad situation. The conversation covers:- What is an edge case?- What humans perceive as edge cases can be different from what machines are struggling with.- Edge cases being rare doesn’t mean they don’t matter. The risk is not about frequency: it’s about frequency and severity.- Edge cases can also be a combination of ‘normal’ things with special meaning that autonomous vehicles have never seen or been trained before.- Robotaxi companies can get more data, but more data is never going to get rid of enough of the edge cases.- Just as edge cases won’t go away, remote assistants — who are there to handle the edge cases encountered by robotaxis —will not go away, either.- The AV industry is at a critical juncture, as it is being tested how well their autonomous vehicles are prepared to handle edge cases.

This week Phil & Junko talk about what it means for Autonomous Vehicle to be safe enough.
At a high level, in addition to net statistical safety, acceptable safety also requires an absence of risk hot spots. It’s not about perfection; it’s about acceptably safe:

• As safe as a human driver on a net statistical basis (accounting for which car, which driver, which road, etc.)

• Avoiding reckless driving

• Avoiding shifting risk onto vulnerable populations

• Absence of unreasonable risk (AUR)

• Standards conformance

• Avoiding negative externalities (e.g., not blocking fire trucks)

• Accountability for breaking road rules

• Meeting all stakeholder safety requirements

A single-metric approach does not work due to multiple stakeholder concerns. A multi-constraint satisfaction approach is the way to go. The bottom line: acceptable safety is meeting the threshold requirements of all your stakeholders.

This episode is a follow-on to Phil’s written piece on How Safe Is Really “Safe Enough” for Autonomous Vehicles?

In this episode we talk with Kristin Poland, Deputy Director, Office of Highway Safety, National Transportation Safety Board (NTSB).

We cover safety issues with Level 2 supervised automation vehicles in general. The NTSB’s recent findings on a pair of Ford BlueCruise crashes motivate a discussion that covers lidar, false alarms, performance limitations, alcohol impairment, telematics, automated emergency braking (AEB), driver monitoring, system integration issues, and the role that NTSB plays in improving transportation safety.

This episode is a follow-on to Phil’s written piece on Level 2 Automated Vehicle Safety.

Mr. Otto Nobetter, meet Mr. Jay Walker

Historian Peter Norton joins Phil & Junko in a discussion of the surprisingly rich history of the slogan "Safety First."

We discuss the history of multiple cycles of adoption and repudiation of the concept of Safety First in industrial and automotive safety. We also cover cautionary tales from the history of safety regulations in other industries. Perhaps it is time for the autonomous vehicle industry to re-think its Safety First messaging.

Substack episode info: https://open.substack.com/pub/avsafety/p/podcast-the-evolution-of-safety-first

Phil & Junko on AV Safety / S1 ep01

Accountability of Remote Assistants