Structured expert judgement is a well-established approach for minimizing biases and encouraging well-informed and thoughtful estimation when working with experts. The appropriate application of this method depends on several factors related to uncertainty, legitimation, and resources, as discussed by Tim Bedford in a CHDS seminar. Bedford is Professor of Decision and Risk Analysis at the University of Strathclyde, Scotland.
There are several approaches to eliciting expert judgments, ranging from Delphi methods to Roger Cooke’s Classical Model to the Sheffield Elicitation Framework to Philip Tetlock’s Good Judgement Project. While guidelines for good practices exist, Bedford notes that there is a surprising lack of discussion around how one selects an appropriate approach for a particular context.
Bedford notes that the first factor to consider is the degree of uncertainty about the quantity to be estimated. Uncertainty may range from a lack of relevant data, conceptual models or heuristics to explanatory models with good predictive power across all relevant contexts, with several intermediate levels. The second factor relates to legitimation; the method must be acceptable and convincing to influential stakeholders. Considerations include evidence of the expertise of the individuals involved and their predictive capability. The third factor is resources, including time, funding, and availability of expertise. Any one of these factors can constrain the way in structured expert judgement may be implemented as well as the quality and validity of the results.
Bedford then discussed an example, focusing on the likelihood of a need for a “black start” of the electric grid after a complete shutdown. In that case, it did not make sense to rely on expert judgement alone. The researchers built a high-level risk model, tested with grid experts, and then ran an expert workshop with grid operators. For the expert judgement protocol, the researchers follow the IDEA approach (“Investigate,” “Discuss,” “Estimate,” “Aggregate”).
Learn more: Read the publication, Reference Case Methods for Expert Elicitation in Healthcare Decision Making
Learn more: Read the publication, Developing a Reference Protocol for Structured Expert Elicitation in Health-Care Decision-Making
Learn more: Read the publication, Introduction to the Special Series on Research Synthesis: A Cross-Disciplinary Approach
Related news: In Memoriam: Daniel Kahneman
Related news: Wright Promoted to Associate Professor