ISPOR Working Group on Generative AI – Center for Health Decision Science

CHDS faculty Jagpreet Chhatwal is member of a newly formed ISPOR Working Group on Generative Artificial Intelligence (GenAI), which published its first report on opportunities, challenges, and policy considerations for GenAI use in health technology assessment. They reviewed applications of GenAI to three areas of research: systematic literature reviews, real-world evidence, and health economic modeling.

There are several ways that GenAI can assist in systematic literature reviews. It can propose search terms, screen text, propose code for meta-analyses and generate drafts. Additionally, GenAI can explain its reasoning for including or excluding articles. However, its accuracy and relevance are dependent on highly reliable prompts, including precise inputs. There were potential errors in classifying abstracts or extracting data and instances of incorrect output. It is necessary to manually verify the terms proposed by the models and continuous human oversight and validation is necessary.

GenAI has the potential to increase data processing efficiency and analysis of real-world evidence and to extract evidence from unstructured notes in health records. However, studies of GenAI models have shown some to be inaccurate in matching descriptive text to the correct disease classifications or procedural codes. The report also mentioned that GenAI can increase bias from the studies on which it is trained and also carries the risk of re-identifying de-identified patient information by memorizing the data on which it was trained.

In modeling, GenAI can add efficiencies in different steps of model building such as conceptualization and parameterization. However, human involvement is needed to guide GenAI to understand the nuances of health economic modeling.

The Working Group sees potential in the use of GenAI in health technology assessment but foresees it in the early stages. The report also noted that the technology is advancing rapidly and the actual performance of GenAI models will improve substantially in the near future. The Working Group will continue to publish reports to educate research community and guide GenAI’s use in health technology assessment.

Learn more: Read the full report, Generative Artificial Intelligence for Health Technology Assessment: Opportunities, Challenges, and Policy Considerations: An ISPOR working group report
Learn more: Read the ISPOR press release, Generative AI Set to Reshape Health Technology Assessment, ISPOR Report Finds