Faces of Digital Health

How will health AI assurance labs look like and who will pay for assessments?

Listen Later

Several organizations are thinking about the right way to regulate AI and the idea of assurance labs which would test and validate AI solutions in the US healthcare is taking shape. This was the topic we discussed with Brian Anderson - CEO of the coalition for Health AI or CHAI: how will assurance labs look like, how much will assessments cost, who will pay for them, and how will AI “nutrition labels” look like. 


Summary:

Assurance Labs in Healthcare AI
  • The Coalition for Health AI (CHI) is developing a network of quality assurance labs to evaluate AI models in healthcare.
  • These labs aim to provide independent, transparent assessments of AI models' performance across different populations.
  • By the end of 2024, CHI plans to have two certified labs operational, with more to follow in 2025.
    • Model Cards and Evaluation
    • CHAI has introduced "model cards" or "nutrition labels" for AI models, describing their training data, methodology, indications, and limitations.
    • Model cards are created by developers, while assurance labs provide independent evaluation reports.
    • CHAI is working on technical specifications for model cards to ensure consistency and transparency.
      • Goals and Benefits
      • Assurance labs aim to balance innovation with safety in AI development.
      • They can help identify model performance issues across different populations and accelerate improvements.
      • The process is intended to build trust in AI among healthcare providers and patients.
        • Implementation and Challenges
        • CHAI is creating a competitive marketplace of quality assurance labs to keep costs reasonable.
        • Labs must be free from conflicts of interest with AI vendors.
        • Evaluation reports will be published in a public registry for transparency.
        • The cost of evaluations is expected to be in the range of thousands of dollars, not millions.
          • Future Plans
          • CHAI is exploring partnerships with health systems and NGOs to establish quality assurance labs in the EU.
          • The initiative aims to be scalable and adaptable to different geographic regions and populations.


            • www.facesofdigitalhealth.com

            Newsletter: https://fodh.substack.com/

            ...more
            View all episodesView all episodes
            Download on the App Store
            Faces of Digital HealthBy Tjasa Zajc
            • 5
            • 5
            • 5
            • 5
            • 5

            5

            20 ratings

            More shows like Faces of Digital Health

            View all
            Freakonomics Radio by Freakonomics Radio + Stitcher

            Freakonomics Radio

            32,182 Listeners

            The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch by Harry Stebbings

            The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch

            527 Listeners

            JAMA Clinical Reviews by JAMA Network

            JAMA Clinical Reviews

            502 Listeners

            Pivot by New York Magazine

            Pivot

            9,538 Listeners

            Negotiate Anything by Kwame Christian Esq., M.A.

            Negotiate Anything

            701 Listeners

            The Peter Attia Drive by Peter Attia, MD

            The Peter Attia Drive

            8,051 Listeners

            FT News Briefing by Financial Times

            FT News Briefing

            656 Listeners

            All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

            All-In with Chamath, Jason, Sacks & Friedberg

            9,633 Listeners

            SmartLess by Jason Bateman, Sean Hayes, Will Arnett

            SmartLess

            58,104 Listeners

            Huberman Lab by Scicomm Media

            Huberman Lab

            29,388 Listeners

            The Ezra Klein Show by New York Times Opinion

            The Ezra Klein Show

            16,007 Listeners

            The Heart of Healthcare | A Digital Health Podcast by Massively Better Healthcare

            The Heart of Healthcare | A Digital Health Podcast

            156 Listeners

            ZOE Science & Nutrition by ZOE

            ZOE Science & Nutrition

            2,083 Listeners