Sign up to save your podcasts
Or
Share Measuring AI with SNOMED CT: Inside the Entity Linking Benchmark
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Measuring AI with SNOMED CT: Inside the Entity Linking Benchmark
Episode 08: Show Notes
Artificial intelligence (AI) is rapidly being adopted across healthcare, with a growing number of tools claiming to work with SNOMED CT. But until now, there has been no standardized way to independently evaluate those claims or assess how effectively AI systems can map clinical information to SNOMED CT concepts.
In this episode, Rory Davidson, Chief Digital Information Officer at SNOMED International, is joined by Will Hardman, Machine Learning and Data Science Specialist at Veritai, and Chris Kucharczyk, Lead Data Scientist at DrivenData, to discuss the launch of the SNOMED CT Entity Linking Benchmark.
Together, they explore why SNOMED International developed the benchmark, how it builds on the earlier SNOMED CT Entity Linking Challenge, and why ongoing evaluation is essential in an era of rapidly evolving AI models. The discussion covers the complex process of building and annotating a clinically credible dataset, the challenges of measuring AI performance in healthcare, and the role the benchmark can play in helping health systems, vendors, and developers better understand the strengths and limitations of their solutions.
Key Points from This Episode:
Quotes:
“But I suppose the real learning from this is that if you have a diverse team, even with a really good set of guidelines, there's still going to be some disagreement over the meaning of terms, and that kind of puts an upper bound on how well you can expect a machine to do. Because if two people only agree 85% of the time, you can't expect a machine to do better than that.” — Will Hardman [09:30]:
“And that is why, if for no other reason than for the interchange of precise information between language models, the terminology has become even more important now than it was before.” — Will Hardman [16:27]:
“Instead of saying, ‘Ah, we are great at this thing’ without an easy way to verify it, now we have something where we can directly compare different approaches, and that interoperability is going to make a big difference in how we're able to evaluate how well these different tools work.” — Chris Kucharczyk [21:19]
“And I think my hope for a success story 12 to 18 months from now is we get to a point where we have good enough systems that can start to bootstrap the next evaluation dataset for a new benchmark.” — Chris Kucharczyk [23:09]
Links Mentioned in This Episode:
Will Hardman- https://www.linkedin.com/in/will-hardman-81898323/
Veritai - https://www.veratai.co.uk/
Chris Kucharczyk - https://www.linkedin.com/in/chris-kucharczyk-9783b937/
DrivenData - https://drivendata.co/
SNOMED CT Entity Linking Benchmark - https://www.drivendata.org/
Rory Davidson - https://www.linkedin.com/in/rorydavidson/
SNOMED International - https://www.snomed.org/
View all episodes
By SNOMED InternationalEpisode 08: Show Notes
Artificial intelligence (AI) is rapidly being adopted across healthcare, with a growing number of tools claiming to work with SNOMED CT. But until now, there has been no standardized way to independently evaluate those claims or assess how effectively AI systems can map clinical information to SNOMED CT concepts.
In this episode, Rory Davidson, Chief Digital Information Officer at SNOMED International, is joined by Will Hardman, Machine Learning and Data Science Specialist at Veritai, and Chris Kucharczyk, Lead Data Scientist at DrivenData, to discuss the launch of the SNOMED CT Entity Linking Benchmark.
Together, they explore why SNOMED International developed the benchmark, how it builds on the earlier SNOMED CT Entity Linking Challenge, and why ongoing evaluation is essential in an era of rapidly evolving AI models. The discussion covers the complex process of building and annotating a clinically credible dataset, the challenges of measuring AI performance in healthcare, and the role the benchmark can play in helping health systems, vendors, and developers better understand the strengths and limitations of their solutions.
Key Points from This Episode:
Quotes:
“But I suppose the real learning from this is that if you have a diverse team, even with a really good set of guidelines, there's still going to be some disagreement over the meaning of terms, and that kind of puts an upper bound on how well you can expect a machine to do. Because if two people only agree 85% of the time, you can't expect a machine to do better than that.” — Will Hardman [09:30]:
“And that is why, if for no other reason than for the interchange of precise information between language models, the terminology has become even more important now than it was before.” — Will Hardman [16:27]:
“Instead of saying, ‘Ah, we are great at this thing’ without an easy way to verify it, now we have something where we can directly compare different approaches, and that interoperability is going to make a big difference in how we're able to evaluate how well these different tools work.” — Chris Kucharczyk [21:19]
“And I think my hope for a success story 12 to 18 months from now is we get to a point where we have good enough systems that can start to bootstrap the next evaluation dataset for a new benchmark.” — Chris Kucharczyk [23:09]
Links Mentioned in This Episode:
Will Hardman- https://www.linkedin.com/in/will-hardman-81898323/
Veritai - https://www.veratai.co.uk/
Chris Kucharczyk - https://www.linkedin.com/in/chris-kucharczyk-9783b937/
DrivenData - https://drivendata.co/
SNOMED CT Entity Linking Benchmark - https://www.drivendata.org/
Rory Davidson - https://www.linkedin.com/in/rorydavidson/
SNOMED International - https://www.snomed.org/