AI - האיגוד הישראלי לרפואה דחופה

How do patients feel about the quality of AI-generated responses to their messages to clinicians?

Author Eleni Linos, MD, DrPH, of Stanford joins JAMA+ AI Editor in Chief Roy H. Perlis, MD, MSc, to discuss her recent study in JAMA Network Open that characterized satisfaction with these messages.

The post JAMA PODCAST: Patient Satisfaction With AI-Generated Responses first appeared on האיגוד הישראלי לרפואה דחופה.

In January, OpenAI, developer of ChatGPT, launched ChatGPT Health, one of many patient-facing generative artificial intelligence (AI) tools in various stages of development.

In January, OpenAI, developer of ChatGPT, launched ChatGPT Health.

From educating patients on women’s sexual health and hip replacement surgery to generating postoperative instructions and digitizing informed consent, the potential medical applications of generative AI tools for the public are vast. In general, their goal is to increase patients’ comprehension of complex medical information and, in the case of ChatGPT Health, provide personalized information based on individual users’ own data. In the not-too-distant future, some experts predict new AI technologies will be able to independently make decisions about patient care.

At their most sophisticated, though, these technologies should serve as a “clinician extender,” not a clinician replacer, said cardiologist Haider Warraich, MD, a program manager at the US government’s Advanced Research Projects Agency for Health (ARPA-H) who previously helped shape digital health and AI policy at the US Food and Drug Administration (FDA).

“I hate the term AI doctor,” Warraich said. “There’s a lot more to me than what these technologies can do.”

There’s more than one reason why using an AI chatbot for health advice is not the same as consulting a physician. Recent studies have raised questions about the accuracy of health information provided by chatbots, and physicians and consumers have expressed concerns over the sharing of personal medical data with large language models (LLMs) that aren’t covered by the Health Insurance Portability and Accountability Act (HIPAA).

ChatGPT Health failed to properly triage the most and the least serious cases in what might be the first study to assess the new tool’s performance, according to an accelerated preview of the article published in late February. The authors, who tested the chatbot using vignettes written by physicians, noted that under-triage of emergency conditions may delay or preclude lifesaving treatment, while over-triage of nonurgent presentations may increase health care utilization.

But LLMs hold promise as a way of expanding access to medical expertise or, at the very least, preparing patients to make the best use of visits with their physicians. “There’s a reason patients want to use these models,” said radiation oncologist Danielle Bitterman, MD, clinical lead for data science and AI at Mass General Brigham. “It’s so hard to access health care right now.”

Of the 800 million users of ChatGPT each week, 1 in 4 seek health-related information, according to Nate Gross, MD, MBA, who leads health care strategy at OpenAI, which developed the chatbot.

“We said, ‘Hey, let’s build some differences to the product to make it a more contextually aware experience,’” as well as one with additional privacy and security connections, he recalled.

Users of “vanilla” ChatGPT, as Gross describes the forerunner of ChatGPT Health, can upload a physician’s note or copy laboratory results from their patient portal, he explained, but those bits of information lack context. “Just uploading a really short doctor’s note could be interpreted very differently if you’re age 20 or age 70.”

ChatGPT Health, on the other hand, invites users to upload all their personal health information, including laboratory test and imaging results as well as data collected by their Apple watch.

Although OpenAI consulted with hundreds of physicians from around the world to improve its models, ChatGPT Health is not designed to play doctor, Gross emphasized.

“We train our models specifically to guide patients to health care professionals for diagnosis and treatment,” he said. “We’re looking to give people information, not tell them if they’re sick, not tell them if they’re healthy. We’re a partner to the health care system in that regard.”

By late February, ChatGPT Health was not yet available to all comers; prospective users could add their name to a waitlist for using the chatbot. OpenAI declined to say how many people have used ChatGPT Health so far.

Privacy is one of users’ main concerns about ChatGPT Health and other LLMs that allow people to upload personal health information.

Elon Musk recently suggested in an X post that “[y]ou can just take a picture of your medical data or upload the file to get a second opinion from Grok,” an AI chatbot developed by his company, xAI.

Commenters were aghast at the idea. One decided to ask Grok’s opinion and posted its reply: “Grok is not HIPAA compliant, and we strongly advise against uploading sensitive medical data.”

Gross acknowledged that ChatGPT Health isn’t HIPAA compliant either. That’s not due to negligence, he pointed out, but because ChatGPT Health, like Grok, is not an entity covered by HIPAA, such as a physician or health insurance plan, or a business associate of a covered entity.

“They are not held to the same legal requirements that doctors and health care institutions are,” Bitterman said of the AI companies.

ChatGPT Health “is building on a lot of very proprivacy protections that ChatGPT already had, with additional layers of protection,” Gross said. “We wanted to set a really high bar.” For example, he noted, OpenAI will not include any ChatGPT Health conversations among the data it uses to train the LLM. And, he explained, as with ChatGPT, ChatGPT Health users can opt to make chats temporary, meaning they won’t appear in their history and ChatGPT Health won’t save them.

Even so, “those assurances may not be worth that much if companies get sold,” pointed out David Liebovitz, MD, codirector of the Institute for Artificial Intelligence in Medicine’s Center for Medical Education in Data Science and Digital Health at the Northwestern University Feinberg School of Medicine.

For now, he said, if patients asked him whether he thought they should try ChatGPT Health, he’d probably suggest “they could wait a little bit longer, when there could be more privacy-related tools.”

Even if they want to, chatbot users—especially clinically challenging patients with long, complex medical histories—can’t always upload all their medical records, Bitterman pointed out.

“It’s very hard to ensure that you have all your medical records,” she said. “Those are the missing pieces that make clinical practice hard.”

Gross acknowledged that “our health care system is very fragmented.” But, he said, if patients forget to upload records from a particular physician or hospital, their physicians’ most recent notes likely will at least mention them.

Even patients who have all the relevant information may not paint a complete picture of their situation when interacting with LLMs, concluded research published in February.

The study, led by the Oxford Internet Institute in the UK, tested whether LLMs could help individuals without medical training identify underlying conditions and choose a course of action in 10 physician-drafted health scenarios. Researchers randomly assigned 1300 participants to receive assistance from 1 of 3 LLMs or, to serve as the control, a source of their choice, which was typically Google. The 3 LLMs were ChatGPT-4o, Meta’s Llama 3, and Command R+, which was developed by Cohere, a Canada-based company.

On average, when the scientists presented the vignettes directly to the LLMs, bypassing human interaction, the chatbots correctly identified the condition 95% of the time and the appropriate course of action 56% of the time.

But when study participants presented the vignettes to the same LLMs, the chatbots correctly identified relevant conditions only about a third of the time and the appropriate course of action less than 44% of the time. The LLMs performed no better than Google did in the control group.

“The limiting factor wasn’t just the model’s medical knowledge,” coauthor Rebecca Payne, MBBS, PhD, MPH, a general practitioner at the North Wales Medical School, Bangor University, said in an email. “It was the human-AI communication loop: people providing incomplete information, the model misinterpreting key details, and, importantly, people failing to carry forward a relevant diagnostic suggestion that the model did raise during the exchange.”

Whether using an LLM or Google, study participants “tended to underestimate the severity in the vignettes we tested,” Payne said. “That raises the risk that some users may feel falsely reassured or may delay seeking care.”

Payne’s findings didn’t surprise Bitterman. “With these chatbots, it’s incumbent on the user to know what they need to provide to the model to get the best information,” she said. “Having that kind of clinical nuance requires a lot of on-the-ground training,” not just the LLMs’ training on medical literature and textbooks.

The advice she gives to patients: “Don’t take immediate action just based on what you find online. We can discuss it together.”

The result could be deadly if, say, a chatbot mistakenly told a user that they didn’t need to go the emergency department because their chest pain was due to indigestion, not a heart attack.

That’s why Payne advises patients to use chatbots only for low-stakes support, such as explaining medical terms, preparing questions for a clinician, and summarizing what they’ve been told. “LLMs currently perform best as ‘assistants/secretaries’ that help organize known information rather than generate high-stakes clinical interpretations,” she said.

Physicians are working on a number of generative AI applications for more focused, lower-stake purposes.

For urologist Gio Cacciamani, MD, the diagnosis of a loved one with a serious disease unrelated to his specialty gave him a taste of what patients face when trying to decipher scientific information.

“When it comes to something outside my field, it’s very challenging to read,” said Cacciamani, director of the Artificial Intelligence Center for Surgical and Clinical Applications in Urology at USC’s Keck School of Medicine. “That situation opened my eyes.”

Cacciamani discovered 2 types of medical information online—either “extremely readable but not certified,” such as blog posts, or “peer-reviewed, certified, but not readable at all,” mainly publications in scientific journals.

Generative AI “has the potential to bridge long-standing gaps between certified medical knowledge and patient understanding,” Cacciamani and coauthors noted in a commentary published in February.

Using the retrieval-augmented generation, or RAG, technique, which trains the LLM with a medically verified knowledge base, he developed a new tool that can translate and summarize abstracts and full articles. More than 6000 people have turned to Pub2Post, and some medical journals are using it for their social media posts, Cacciamani said.

Antonio Forte, MD, a plastic surgeon at the Mayo Clinic in Jacksonville, Florida, used RAG to develop an LLM virtual assistant for postoperative instructions.

Patients often are discharged after surgery while still experiencing the residual effects of anesthesia or painkillers, making it difficult to remember postoperative instructions, Forte said. And, he added, they frequently misplace printouts of the information. “That’s why we thought, ‘What if we got patients the ability 24/7 to have access to high-quality, medically verified information?’”

Using simulated patient interactions, testing the virtual assistant demonstrated strong technical accuracy, safety, and clinical relevance, albeit at a relatively high 11th-grade reading level, Forte and his coauthors recently reported.

And Bitterman has tested the ability of ChatGPT-4o and Llama 3.2-8B to answer patients’ questions about clinical trials with the goal of simplifying informed consent forms. In a recent study, she and her coauthors found that ChatGPT-4o was significantly more reliable and safer that Llama 3.2-8B in answering these queries.

In January, 2 federal agencies, both part of the US Department of Health and Human Services, launched initiatives focusing on digital health tools for patients with common, chronic conditions. One is designed to evaluate a regulatory pathway for digital health tools including LLMs, and the other aims to spur the development of an LLM for patients with heart failure.

The FDA, working with the Center for Medicare & Medicaid Innovation, announced the Technology-Enabled Meaningful Patient Outcomes (TEMPO) for Digital Health Devices Pilot.

According to the FDA, the voluntary pilot will evaluate a new enforcement approach “that supports digital health devices intended for use to improve patient outcomes in cardio-kidney-metabolic, musculoskeletal, and behavioral health conditions.”

The FDA has not yet authorized any LLM, Warraich said. Generative AI applications such as LLMs “present a unique challenge because of the potential for unforeseen, emergent consequences,” according to a Special Communication he coauthored in JAMA in 2024.

Today, Warraich is leading a new ARPA-H initiative whose goal is the development of new LLM systems that are ready for submission to the FDA within 2 years for authorization as medical devices. The Agentic AI-Enabled Cardiovascular Care Transformation (ADVOCATE) program “aims to transform advanced cardiovascular disease management with an agentic AI system that can provide 24/7 holistic clinical care.”

“I believe that as AI presents an opportunity to fundamentally transform what it means to be a clinician, a patient, and the relationship between them, cardiology will be at the tip of the spear…,” Warraich noted in an opinion piece published in February in the Journal of the American College of Cardiology.

The first use for technologies developed through ADVOCATE will be providing care for patients with congestive heart failure. If a patient is feeling short of breath, for example, the technology will decide if the patient should go to the emergency department and whether they might need a new prescription or a higher dose of a current medication, Warraich explained. Along with developing AI agents that can be trusted to make such changes autonomously, ADVOCATE will also support the creation of a supervisory AI “overseer” to monitor the safety and effectiveness of clinical AI agents after they’ve been deployed by health systems.

Given that ChatGPT is only 3 years old, the rapid development of new generative AI applications for patient use may seem like science fiction. As Bitterman said, “This is so far beyond what I would have predicted 5 years ago.”

Published Online: March 6, 2026. doi:10.1001/jama.2026.1122

Conflict of Interest Disclosures: Dr Bitterman reported serving as an associate editor of JCO Clinical Cancer Informatics, Annals of Oncology, and radiation oncology for HemOnc.org. She also reported receiving consulting fees from Inspire Exercise Medicine LLC and honoraria from Harvard Medical School, Med-IQ, and the National Comprehensive Cancer Network and serving as a scientific advisory board member for Blue Clay Health LLC and Mercurial AI. Dr Liebovitz reported receiving research grants from Children’s Hospital of Philadelphia, the FDA, Merck Sharp & Dohme, the National Institutes of Health, the National Science Foundation, and the University of Chicago. He also reported that he has an ownership or investment interest in CodeAccelerate, Dendritic Health AI, KYRAL Inc, and Optima Integrated Health Inc. Dr Cacciamani reported holding equity in EditorAIPro, of which Pub2Post is a product. Dr Forte reported that his research at Mayo has been funded by Dalio Philanthropies, the Gerstner Family Foundation, the Richard M. Schulze Family Foundation, and Schmidt Sciences and that he is a paid medical advisor for OpenEvidence. No other disclosures were reported.

The post JAMA PODCAST: Are AI Tools Ready to Answer Patients’ Questions About Their Medical Care? first appeared on האיגוד הישראלי לרפואה דחופה.

Date: January 3, 2026 

Reference: Shroyer et al. Accuracy of cath lab activation decisions for STEMI-equivalent and mimic ECGs: Physicians vs. AI (Queen of Hearts by PMcardio). Am J Emerg Med. 2025 Nov.

Guest Skeptic: Dr. Amal Mattu has been on the faculty at the University of Maryland since 1996. He has developed an academic niche in emergency cardiology and electrocardiography, and he also enjoys teaching and writing on other topics, including emergency geriatrics, faculty development, and risk management. Amal is currently a tenured professor and Vice Chair of Emergency Medicine at the University of Maryland School of Medicine, and a Distinguished Professor of the University of Maryland-Baltimore.

Case: A 58-year-old man with diabetes and hypertension arrives at the emergency department (ED) 30 minutes after the sudden onset of substernal chest pressure radiating to the left arm, now improved to 3/10. His vital signs are BP 146/88, HR 92, RR 18, O2 sat 98% on room air. The initial 12-lead ECG shows RBBB with left anterior fascicular block and subtle anterior ST‑depression with proportionally tall, broad T waves in V2 to V4. This is an appearance that can be seen with Hyper-Acute T Wave Occlusive Myocardial Infarction (HATW‑OMI) or an ST-Elevated Myocardial Infarction (STEMI)‑mimic in conduction disease. A debate ensues between emergency medicine and cardiology on whether to activate the cath lab now or get troponins plus serial ECGs?

Background: Emergency physicians need to be experts at interpreting ECGs. For decades, we’ve been taught STEMI criteria, only to learn repeatedly that important exceptions exist (posterior OMI, de Winter, hyperacute T waves, modified Sgarbossa in LBBB, etc.). Those exceptions have evolved into two distinct categories. There are the STEMI‑equivalents (OMI without classic ST‑elevation) and STEMI‑mimics (ST‑elevation without OMI). That expanding exception list increases diagnostic complexity and uncertainty. This is the area where artificial intelligence (AI), utilizing computer vision and machine learning, could provide a benefit.

ECG-specific AI models now aim squarely at this problem. The study we are reviewing today evaluated the Queen of Hearts (QoH) AI. It is a deep neural network trained to detect occlusive myocardial infarction (OMI) on 12-lead ECGs. The model is described as “91% accurate” in prior work and is undergoing FDA review as of March 24, 2025, but whether it outperforms practicing clinicians on the hardest cases (STEMI‑equivalents and mimics) remained unclear.

ECG diagnostic accuracy is important in emergency medicine because misclassification cuts both ways. Missed OMI delays reperfusion, while overcalls send patients and teams to the cath lab unnecessarily, putting patients at risk and using up valuable resources. A diagnostic aid that catches true positive OMIs while reducing false activations could improve outcomes and team throughput.

Reference: Shroyer et al. Accuracy of cath lab activation decisions for STEMI-equivalent and mimic ECGs: Physicians vs. AI (Queen of Hearts by PMcardio). Am J Emerg Med. 2025 Nov.

Authors’ Conclusions: “Physicians frequently misinterpret STEMI-equivalent and STEMI-mimic ECGs, potentially impacting CLA decisions. QoH AI demonstrated superior accuracy, suggesting a potential to reduce missed OMIs and unnecessary catheterization laboratory activations. Prospective studies are needed to validate these findings in clinical practice.” 

Quality Checklist for a Diagnostic Study:

Results: They recruited 95 physicians to interpret the ECGs. There were 53 EM physicians and 42 cardiologists (23 general, 15 interventional, 4 EP electrophysiology). Experience: EPs 7 years (IQR 3 to 15) vs cardiologists 15 years (IQR 9.2 to 21).

The most frequently misclassified by humans were LBBB (±OMI), transient STEMI, HATW‑OMI, and de Winter. QoH AI missed LBBB‑OMI and LV aneurysm. RBBB + fascicular block and HATW‑OMI produced the largest EP-cardiologist disagreement.

1) Spectrum Bias: The investigators intentionally selected “ambiguous” STEMI‑equivalent and STEMI‑mimic ECGs and fixed the OMI prevalence at 50% for the reader study. That design improves efficiency in comparing readers and the AI, but it does not reflect the spectrum or prevalence we see in day-to-day ED practice and therefore threatens external validity. In diagnostic accuracy research, spectrum bias occurs when the distribution of disease/non-disease, disease severity, or look-alikes in the sample differs from that in the clinical population in which the test will be used. It can change sensitivity and specificity in either direction. Selecting borderline cases may deflate both compared with routine practice, and it will certainly distort PPV/NPV because predictive values are prevalence‑dependent. The authors acknowledge this by noting the 50% OMI prevalence and the deliberate use of ambiguous ECGs “may not accurately reflect predictive values observed in real-world settings.”

2) Differential Verification & Imperfect Gold Standard: Not every patient had the same reference standard. While most OMI determinations used angiography, some mimic cases without angiography were adjudicated by serial troponins, echocardiography, and clinical follow-up. Using different reference standards in different subgroups constitutes differential verification (double gold‑standard) bias and can bias sensitivity and specificity up or down, depending on whether the disease can resolve or only become detectable over time. In addition, any composite or clinical adjudication process is an imperfect gold standard, which can either inflate or deflate the index test’s performance depending on how errors correlate across tests. The authors explicitly note these issues in their discussion.

3) Incorporation/Review Bias: The paper reports that cardiologists performing angiography were not masked to the ECG. When the result of (or information from) the index test helps determine the reference diagnosis, that is incorporation (review) bias. This typically inflates both sensitivity and specificity of the index test because the gold standard classification is partially “contaminated” by the test under study. In this context, seeing a concerning ECG may tilt the invasive assessment and adjudication toward “culprit” lesion labelling or influence borderline calls, making ECG-based classification look better than it truly is.

4) Unit‑of‑analysis & Precision Limitations: This was a reader study with 95 clinicians classifying the same small set of 18 ECGs. Even with appropriate statistics, the small number of cases means performance estimates can be fragile, and the 95% confidence intervals reflect that imprecision. To their credit, the authors modelled accuracy with multi-level robust variance to account for clustering (multiple readers rating the same cases), but the design still limits precision and generalizability across the full morphology spectrum of each category. The authors themselves state that “one representative ECG per type…cannot represent all ST‑T variants”, and that asking physicians to read far more than 18 tracings was impractical. This imprecision concerns should raise our skeptical radar, and we should factor this into our study interpretation.

5) External Validity: The study is single-center and uses an online survey without the interruptions, time pressure, serial ECGs, bedside echo, or troponin trends that influence ED decision‑making. The authors explicitly caution that the controlled survey conditions do not replicate real clinical environments and could over- or under-estimate real-world accuracy. AI performance can also be domain‑dependent (ECG device/process, patient mix). Showing “simulation” superiority does not guarantee clinical utility until confirmed in prospective practice studies. This is a limitation known in the diagnostic literature (including the AI diagnostic literature) that emphasizes the potential difference between an artificial scenario and prospective bedside clinical application. In other words, until it is released into the wild involving other hospitals and workflows, we don’t know if it will have a net positive patient-oriented outcome (POO) of benefit.

Comment on the Authors’ Conclusion Compared to the SGEM Conclusion: We generally agree with the authors’ conclusions.

Case Resolution: Given the concerning RBBB + LAFB with anterior repolarization changes and ongoing symptoms, we activate the cath lab. If available, an AI read supporting OMI would reinforce the call. If it disagreed, we would not delay for the algorithm. In the lab, the patient is found to have a proximal LAD culprit and undergoes PCI.

Dr. Amal Mattu

Clinical Application: The most significant finding that immediately makes me doubt the utility of the study is the chart that shows sensitivity and specificity. Even without all of the nerdy details, it is obvious that a 65% sensitivity for picking up OMIs (or my long-preferred term: acute coronary occlusion, ACO) is not possible. Any emerg phys or cardiologist who is missing 35% of ACOs is going to be fired, sued many times over, and driven out of medicine! Without reading any of the paper, as soon as I see that chart, I know it can’t be right unless the clinicians at his institution are incompetent.

No one has discussed the costs associated with integrating AI systems into all those ECG machines.

This QoH AI system is not ready for implementation in clinical practice outside a research study. The use of AI has the potential to augment clinical decision-making, but it is not currently done on autopilot without a human-in-the-loop.

What Do I Tell the Patient? Your ECG shows changes that could mean a blocked heart artery. We think the best course of action is to take you to the cath lab now to restore blood flow if needed. We also use a computer tool to double-check ECGs. It supports our concern, but it doesn’t replace us as doctors. The goal is to act fast and safely to get you the care you need.

Keener Kontest: Last week’s winner was Dr. Steven Stelts from Auckland, NZ. He knew the enzyme inhibited by etomidate to decrease cortisol and aldosterone is 11-beta-hydoxylase.

The post PODCAST: Playing with the Queen of Hearts – AI, Is It Very Smart (for ECG Interpretation)? first appeared on האיגוד הישראלי לרפואה דחופה.

Roy Perlis, MD, MSc1; Yulin Hswen, ScD, MPH2

In this episode of JAMA+ AI Conversations, we review the past year in AI and health, and talk about what we hope to see this coming year. We discuss our favorite articles and what they might mean for the future of medicine.

This interview is part of a series in which JAMA Network editors and expert guests explore issues surrounding the rapidly evolving intersection of AI and medicine.

The post PODCAST: AI and Health—The Year in Review first appeared on האיגוד הישראלי לרפואה דחופה.

Date: October 28, 2024

Reference: Woelfle T et al. Benchmarking Human–AI collaboration for common evidence appraisal tools. J Clin Epi Sept 2024.

Guest Skeptic: Dr. Laura Walker is an Associate Professor of Emergency Medicine and the vice chair for digital emergency medicine at the Mayo Clinic. In addition to finding ways to use technology in emergency department (ED) care, she is interested in how health systems work and how patients move through the healthcare system. Previously, she has been an ED medical director, quality chair, and regional hospital director.

Case: The Mayo Clinic Department of Emergency Medicine is planning its next journal club. It has recently been boosted by adding Dr. Chris Carpenter to the faculty who created the amazing JC at Washington University. A resident has been assigned to report on the PRISMA quality checklist for the SRMA they will discuss. She has been playing around with ChatGPT 3.5 from OpenAI and wonders if it could do this task quickly and, more importantly, accurately.

Background: In recent years, large language models (LLMs), such as GPT-4 and Claude, have shown remarkable potential in automating and improving various aspects of medical research. One intriguing area of exploration is their ability to assist in critical appraisal, a cornerstone of evidence-based medicine (EBM). Critical appraisal involves evaluating the quality, validity, and applicability of studies using structured tools like PRISMA, AMSTAR and PRECIS-2.

Traditionally, these tasks have been manual, often requiring significant expertise and time to ensure accuracy. However, as LLMs have evolved, their ability to interpret and analyze complex textual data presents a unique opportunity to enhance the efficiency of these appraisals. Research into the accuracy of LLMs, when used for appraising clinical trials and systematic reviews, is still in its early stages but holds promise for the future of automated medical literature assessment​​.

Given that assessing the validity and quality of clinical research is essential for ensuring that decisions are based on reliable evidence, this development raises important questions. Can these advanced AI tools perform critical appraisals with the same accuracy and reliability as human experts? More importantly, how can they augment human decision-making in medical research?

Reference: Woelfle T et al. Benchmarking Human–AI collaboration for common evidence appraisal tools. J Clin Epi Sept 2024.

Authors’ Conclusions: “Current LLMs alone appraised evidence worse than humans. Humane-AI collaboration may reduce workload for the second human rater for the assessment of reporting (PRISMA) and methodological rigor (AMSTAR) but not for complex tasks such as PRECIS-2.”

Quality Checklist for a Diagnostic Study:

Results: They assessed 112 SRMAs and 56 RCTs. Humans performed the best reporting the highest accuracy for all three assessment tools. Of the LLMs, Claude-3-Opus consistently performed the best across PRISMA and AMSTAR, indicating that it may be the most reliable LLM for these tasks.

GPT-3.5, despite being an older and smaller model, performed better than newer LLMs like GPT-4 and Claude-3-Opus on the more complex PRECIS-2 tasks.

The collaborative human-AI approach yielded superior performance compared to individual LLMs, with accuracies reaching 96% for PRISMA and 95% for AMSTAR when humans and LLMs worked together.

1. Bias in Training Data and Prompts: LLMs rely on the data they were trained on, which may introduce unseen biases​. In addition, the behaviour of the model is impacted by the information it was fed (prompts). As an example, when the LLMs were required to pull relevant quotes, they did not follow the instructions, often pulling too many or pulling quotes from analyses they had already performed rather than the source material.

2. Limited Contextual Understanding by LLMs: LLMs may lack the nuanced judgment needed for assessing methodological quality in complex trials​. This was illustrated by LLMs having a low accuracy while the addition of a human rater increased the accuracy. LLMs still don’t process in the same way as humans, but when we are the gold standard is that something we want to match or is there a benefit to re-evaluating our responses after the LLM goes through to see if we are making errors? The authors identify a prospective evaluation as a potential next step in understanding how LLMs perform and how to improve assessments

3. Lack of Transparency in LLM Decision Processes: Transparency in decision-making LLMs presents significant challenges. A key issue is the “black box” nature of these systems. This often makes it difficult to explain how they reach their decisions even to experts​. LLMs can generate sophisticated outputs from simple prompts but, the underlying reasons are opaque. AI often misunderstands or simplifies tasks, creating outputs that are sometimes unpredictable and difficult to interpret, further complicating transparency​. This raises concerns about trust in the LLMs results​.

4. Cost, Time and Financial: When the bandwidth is throttled (only so many times you can use it per day and limits on the length of the paper that can be evaluated) and the cost gets high ($115) this limits the utility – and cost/time are balanced with quality with the newer models being costly and the older being less expensive but throttled.

5. Newer Appraisal Tools: The last few years have seen a dramatic acceleration in the development of LLMs, transforming them from experimental tools into mainstream applications. This rapid improvement is primarily driven by advances in neural networks, scaling in model size, and more sophisticated training techniques. LLMs like ChatGPT have achieved unprecedented adoption rates, reaching millions of users within months of release due to their versatility and effectiveness in a wide variety of tasks​. They can now perform more complex problem-solving capabilities, demonstrating learning abilities that were unthinkable just a few years ago​. It is reasonable to expect LLMs to continue to improve and become more accurate in assessing critical appraisal tools.

Comment on Authors’ Conclusion Compared to SGEM Conclusion: We generally agree with the authors’ conclusions.

Case Resolution: You ask the resident for a tutorial on LLMs and use it to evaluate the SRMA adherence to the PRISMA guidelines. That way you will learn how to use some AI for critical appraisal while ensuring the program’s output is accurate based on your expertise.

Dr. Laura Walker

Clinical Application: LLMs for critical appraisal are interesting but not ready for prime time.

What Do I Tell the Resident? Hey, using AI for critical appraisal sounds like a cool idea. I don’t know much about LLMs. Can we do the evaluation together? I would like to learn more about this new technology and ensure it provides the right information for our journal club.

Keener Kontest: Last week’s winner was Dr. Cindy Bitter. She knew a nickname often used to describe the experience of dissociation and hallucinations that can occur with ketamine use is the K-Hole.

Listen to the SGEM podcast for this week’s question. If you know, then send an email to [email protected] with “keener” in the subject line. The first correct answer will receive a shoutout on the next episode.

The post PODCAST: Domo Arigato Misuta Roboto – Using AI to Assess the Quality of the Medical Literature first appeared on האיגוד הישראלי לרפואה דחופה.