More efficiently using machine learning to assess data was the subject of a recent CHDS seminar with Michael Lingzhi Li. Li is an Assistant Professor in the Technology and Operations Management unit at Harvard Business School.
Machine learning can make important contributions to data-driven decision making, which is a center piece of evidence-based medicine. With machine learning, an algorithm is trained on one or more datasets. Decision makers can then use the results from the trained algorithms (also commonly referred as ‘learned policy’) to guide their choices. The quality and the effectiveness of these decisions depend heavily on the performance of the learned policy, however.
In this seminar, Li discussed the challenges with current policy evaluation methods and introduced the ‘cram method.’ In most application settings, before any model can be implemented, its performance needs to be evaluated. There are two primary evaluation methods: sample-splitting and resampling methods. Sample-splitting uses an out-of-sample testing set to evaluate the trained algorithm. This method is simple, generally computationally efficient, and provides point estimators and confidence intervals around the particular trained model. But sample-splitting is data inefficient, especially for small samples, as a testing set needs to be set aside purely for evaluation.
Resampling methods, including bootstrap and cross-validation, offer more data efficient solutions. These methods train multiple models with resampled datasets to utilize the entire dataset provide an estimate that is the average of these many models. However, they do not evaluate the particular model of interest directly.
The cram method developed by Li and his team addresses limitations of these current approaches, promoting more data-efficient policy learning and evaluation. In a single pass of data, it repeatedly trains a machine learning algorithm and tests its empirical performance. Under this method, the analyst randomly divides the dataset into batches. Each batch is then sequentially used to train the algorithm, while the remaining data are used to evaluate performance improvement. After working through all the batches, the performance of the final algorithm can be quantified. Li noted that this method is significantly more data-efficient than the conventional sample-splitting methods because it uses the entire sample for both learning and evaluation, and it can be used for both offline and online policy learning and evaluation. Li discussed settings where data-efficient learning and evaluation algorithms would be most valuable in health decision making, including clinical trials and other data sets that include a relatively small number of individuals.
Learn more: Read the publication, The Cram Method for Efficient Simultaneous Learning and Evaluation
Related news: Fairness of Machine-Assisted Decisions
Related news: 2024 Raiffa Fund Awards