Uncertainty in Model-Based Decision Making – Center for Health Decision Science

Health policy decisions often rely on decision-analytic computer models to compare costs and health outcomes when direct evidence is incomplete. Although we can almost never eliminate uncertainty, we can manage it, as discussed by Mark Strong in a recent CHDS seminar. Strong is Dean of the School of Medicine and Population Health at the University of Sheffield, UK, as well as a public health physician and Professor of Public Health.

Strong grounded the discussion in a familiar policy problem: deciding whether to fund a new drug when evidence is fragmented and indirect. Uncertainty arises from multiple sources (parameter, structural, methodological, and heterogeneity) and cannot be eliminated. Even in a deterministic world, prediction remains infeasible in social systems due to feedback loops, network effects, non-linearity, black swans, and so on. Within Strong’s framework, models are structured expressions of beliefs about parameters, structure, and data-generating processes. Rather than eliminating uncertainty, they synthesize evidence to support decision-making under uncertainty.

Value of Information (VOI) methods quantify the cost of uncertainty. Expected Value of Perfect Information (EVPI) measures the value of eliminating all parameter uncertainty. Expected Value of Partial Perfect Information (EVPPI) identifies which parameters matter most, while Expected Value of Sample Information (EVSI) estimates the value of collecting new data.

The use of nonparametric regression methods makes EVPPI and EVSI computationally feasible. Traditional nested Monte Carlo approaches require thousands of model evaluations and can be computationally prohibitive. Strong showed this by treating conditional expectations as smooth functions and estimating them via regression (e.g., generalized additive models [GAM] or Gaussian processes [GP]), thereby enabling VOI to be computed directly from probabilistic sensitivity analysis samples. This greatly reduces computational burden while maintaining flexibility and avoiding strong parametric assumptions.

However, VOI lives entirely in the “model world” and depends on the analyst’s assumptions. When combined with real-world study costs (ENBS), caution is required. Strong concluded that while models are inherently imperfect, carefully quantified uncertainty can meaningfully inform policy decisions.

Learn more: Read the publication, Estimating Multiparameter Partial Expected Value of Perfect Information from a Probabilistic Sensitivity Analysis Sample: A Nonparametric Regression Approach
Learn more: Read the publication, Estimating the Expected Value of Sample Information Using the Probabilistic Sensitivity Analysis Sample: A Fast, Nonparametric Regression-Based Method
Learn more: Join our CHDS mailing list to be notified about future CHDS seminars.