Translational Evaluation of Interpretable Machine Learning for Cardiovascular Risk Prediction: Calibration Decomposition, Subgroup Audits, and Decision-Utility Analysis

Article Sidebar

PDF
DOI: https://doi.org/10.28924/2291-8639-24-2026-202
Published: Jul 6, 2026
Cite as:
Hathaichanok Chompoopong, Warawut Narkbunnum, Translational Evaluation of Interpretable Machine Learning for Cardiovascular Risk Prediction: Calibration Decomposition, Subgroup Audits, and Decision-Utility Analysis, Int. J. Anal. Appl., 24 (2026), 202.

Main Article Content

Hathaichanok Chompoopong, Warawut Narkbunnum

Abstract

Cardiovascular disease risk prediction models are often evaluated primarily by discrimination, although translational decision making depends on well-calibrated probabilities, subgroup reliability, and demonstrated clinical utility at actionable risk thresholds. This study conducted a translational evaluation of interpretable machine-learning models for heart disease prediction using a deployment-oriented framework integrating discrimination, calibration (including Murphy decomposition), explainability, subgroup stability, and decision-utility analysis via decision curve analysis. Using a large secondary dataset (308,774 observations; 19 predictors; prevalence 8.1%), models were trained with a stratified hold-out design and evaluated on a fixed test set. Histogram-based gradient boosting achieved the strongest discrimination (PR-AUC 0.3177; AUROC 0.8407) and strong probabilistic accuracy (Brier score 0.0633; ECE 0.0045), with Murphy decomposition indicating minimal reliability loss while preserving resolution. Explainability analyses (SHAP with PDP/ALE/ICE diagnostics) enabled transparent assessment of feature contributions and nonlinear effects relevant to plausibility and governance. Subgroup analyses indicated broadly stable discrimination but more variable calibration across age and self-reported general health strata, supporting the need for subgroup-aware monitoring. Decision curve analysis demonstrated positive net benefit relative to treat-all and treat-none strategies across screening-relevant thresholds (0.05–0.15), with workload trade-offs informing threshold selection for practice.

Article Details

References

  1. R.D. Riley, L. Archer, K.I.E. Snell, J. Ensor, P. Dhiman, et al., Evaluation of Clinical Prediction Models (Part 2): How to Undertake an External Validation Study, BMJ 384 (2024), e074820. https://doi.org/10.1136/BMJ-2023-074820.
  2. A.J. Vickers, E.B. Elkin, Decision Curve Analysis: A Novel Method for Evaluating Prediction Models, Med. Decis. Mak. 26 (2006), 565-574. https://doi.org/10.1177/0272989X06295361.
  3. G.S. Collins, K.G.M. Moons, P. Dhiman, R.D. Riley, A.L. Beam, et al., TRIPOD+AI Statement: Updated Guidance for Reporting Clinical Prediction Models That Use Regression or Machine Learning Methods, BMJ 385 (2024), e078378. https://doi.org/10.1136/BMJ-2023-078378.
  4. B. Van Calster, A.J. Vickers, Calibration of Risk Prediction Models, Med. Decis. Mak. 35 (2014), 162-169. https://doi.org/10.1177/0272989X14547233.
  5. B. Van Calster, D.J. McLernon, M. van Smeden, L. Wynants, et al., Calibration: The Achilles Heel of Predictive Analytics, BMC Med. 17 (2019), 230. https://doi.org/10.1186/s12916-019-1466-7.
  6. A.J. Vickers, B. Van Calster, E.W. Steyerberg, Net Benefit Approaches to the Evaluation of Prediction Models, Molecular Markers, and Diagnostic Tests, BMJ 352 (2016), i6. https://doi.org/10.1136/BMJ.I6.
  7. S. Mora, N.K. Wenger, N.R. Cook, J. Liu, B.V. Howard, et al., Evaluation of the Pooled Cohort Risk Equations for Cardiovascular Risk Prediction in a Multiethnic Cohort from the Women’s Health Initiative, JAMA Intern. Med. 178 (2018), 1231-1240. https://doi.org/10.1001/JAMAINTERNMED.2018.2875.
  8. J.S. Rana, G.H. Tabada, M.D. Solomon, J.C. Lo, M.G. Jaffe, et al., Accuracy of the Atherosclerotic Cardiovascular Risk Equation in a Large Contemporary, Multiethnic Population, J. Am. Coll. Cardiol. 67 (2016), 2118-2130. https://doi.org/10.1016/J.JACC.2016.02.055.
  9. C.A. Emdin, A.V. Khera, P. Natarajan, D. Klarin, U. Baber, et al., Evaluation of the Pooled Cohort Equations for Prediction of Cardiovascular Risk in a Contemporary Prospective Cohort, Am. J. Cardiol. 119 (2017), 881-885. https://doi.org/10.1016/j.amjcard.2016.11.042.
  10. B. Van Calster, L. Wynants, J.F. Verbeek, J.Y. Verbakel, E. Christodoulou, et al., Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators, Eur. Urol. 74 (2018), 796-804. https://doi.org/10.1016/J.EURURO.2018.08.038.
  11. K.G. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, et al., PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration, Ann. Intern. Med. 170 (2019), W1-W33. https://doi.org/10.7326/M18-1377.
  12. R.F. Wolff, K.G. Moons, R.D. Riley, P.F. Whiting, M. Westwood, et al., PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies, Ann. Intern. Med. 170 (2019), 51-58. https://doi.org/10.7326/M18-1376.
  13. S.E. Davis, R.A. Greevy, C. Fonnesbeck, T.A. Lasko, C.G. Walsh, et al., A Nonparametric Updating Method to Correct Clinical Prediction Model Drift, J. Am. Med. Inform. Assoc. 26 (2019), 1448-1457. https://doi.org/10.1093/JAMIA/OCZ127.
  14. T.L. Su, T. Jaki, G.L. Hickey, I. Buchan, M. Sperrin, A Review of Statistical Updating Methods for Clinical Prediction Models, Stat. Methods Med. Res. 27 (2016), 185-197. https://doi.org/10.1177/0962280215626466.
  15. C. Yang, J.A. Kors, S. Ioannou, L.H. John, A.F. Markus, et al., Trends in the Conduct and Reporting of Clinical Prediction Model Development and Validation: A Systematic Review, J. Am. Med. Inform. Assoc. 29 (2022), 983-989. https://doi.org/10.1093/JAMIA/OCAC002.
  16. Y. Cai, Y.Q. Cai, L.Y. Tang, Y.H. Wang, M. Gong, et al., Artificial Intelligence in the Risk Prediction Models of Cardiovascular Disease and Development of an Independent Validation Screening Tool: A Systematic Review, BMC Med. 22 (2024), 56. https://doi.org/10.1186/s12916-024-03273-7.
  17. T.P. Debray, Y. Vergouwe, H. Koffijberg, D. Nieboer, E.W. Steyerberg, et al., A New Framework to Enhance the Interpretation of External Validation Studies of Clinical Prediction Models, J. Clin. Epidemiol. 68 (2015), 279-289. https://doi.org/10.1016/j.jclinepi.2014.06.018.
  18. K. Luijken, R.H.H. Groenwold, B. Van Calster, E.W. Steyerberg, M. van Smeden, Impact of Predictor Measurement Heterogeneity Across Settings on the Performance of Prediction Models: A Measurement Error Perspective, Stat. Med. 38 (2019), 3444-3459. https://doi.org/10.1002/SIM.8183.
  19. K. Luijken, L. Wynants, M. van Smeden, B. Van Calster, E.W. Steyerberg, et al., Changing Predictor Measurement Procedures Affected the Performance of Prediction Models in Clinical Examples, J. Clin. Epidemiol. 119 (2020), 7-18. https://doi.org/10.1016/J.JCLINEPI.2019.11.001.
  20. A. Mishra, R.L. McClelland, L.Y.T. Inoue, K.F. Kerr, Recalibration Methods for Improved Clinical Utility of Risk Scores, Med. Decis. Mak. 42 (2021), 500-512. https://doi.org/10.1177/0272989X211044697.
  21. C. Hong, M.J. Pencina, D.M. Wojdyla, J.L. Hall, S.E. Judd, et al., Predictive Accuracy of Stroke Risk Prediction Models Across Black and White Race, Sex, and Age Groups, JAMA 329 (2023), 306-317. https://doi.org/10.1001/JAMA.2022.24683.
  22. K. Chakradeo, I. Huynh, S.B. Balaganeshan, O.L. Dollerup, H. Gade-Jørgensen, et al., Navigating Fairness Aspects of Clinical Prediction Models, BMC Med. 23 (2025), 567. https://doi.org/10.1186/S12916-025-04340-3.
  23. C.G. Walsh, K. Sharman, G. Hripcsak, Beyond Discrimination: A Comparison of Calibration Methods and Clinical Usefulness of Predictive Models of Readmission Risk, J. Biomed. Inform. 76 (2017), 9-18. https://doi.org/10.1016/J.JBI.2017.10.008.
  24. A.J. Vickers, B. van Calster, E.W. Steyerberg, A Simple, Step-By-Step Guide to Interpreting Decision Curve Analysis, Diagn. Progn. Res. 3 (2019), 18. https://doi.org/10.1186/s41512-019-0064-7.
  25. M. Sadatsafavi, A. Adibi, M. Puhan, A. Gershon, S.D. Aaron, et al., Moving Beyond AUC: Decision Curve Analysis for Quantifying Net Benefit of Risk Prediction Models, Eur. Respir. J. 58 (2021), 2101186. https://doi.org/10.1183/13993003.01186-2021.
  26. D. Piovani, R. Sokou, A.G. Tsantes, A.S. Vitello, S. Bonovas, Optimizing Clinical Decision Making with Decision Curve Analysis: Insights for Clinical Investigators, Healthcare 11 (2023), 2244. https://doi.org/10.3390/HEALTHCARE11162244.
  27. A.V. Ponce‐Bobadilla, V. Schmitt, C.S. Maier, S. Mensing, S. Stodtmann, Practical Guide to SHAP Analysis: Explaining Supervised Machine Learning Model Predictions in Drug Development, Clin. Transl. Sci. 17 (2024), e70056. https://doi.org/10.1111/CTS.70056.
  28. Y. Huang, W. Li, F. Macheret, R.A. Gabriel, L. Ohno-Machado, A Tutorial on Calibration Measurements and Calibration Models for Clinical Prediction Models, J. Am. Med. Inform. Assoc. 27 (2020), 621-633. https://doi.org/10.1093/JAMIA/OCZ228.
  29. G.W. Brier, Verification of Forecasts Expressed in Terms of Probability, Mon. Weather. Rev. 78 (1950), 1-3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.
  30. A.H. Murphy, A New Vector Partition of the Probability Score, J. Appl. Meteorol. 12 (1973), 595-600. https://doi.org/10.1175/1520-0450(1973)012<0595:anvpot>2.0.co;2.
  31. S. Siegert, Simplifying and Generalising Murphy's Brier Score Decomposition, Q. J. R. Meteorol. Soc. 143 (2017), 1178-1183. https://doi.org/10.1002/QJ.2985.
  32. E.W. Steyerberg, A.J. Vickers, N.R. Cook, T. Gerds, et al., Assessing the Performance of Prediction Models, Epidemiology 21 (2010), 128-138. https://doi.org/10.1097/EDE.0B013E3181C30FB2.
  33. D.B. Stephenson, C.A.S. Coelho, I.T. Jolliffe, Two Extra Components in the Brier Score Decomposition, Weather. Forecast. 23 (2008), 752-757. https://doi.org/10.1175/2007WAF2006116.1.
  34. T. Dimitriadis, T. Gneiting, A.I. Jordan, Stable Reliability Diagrams for Probabilistic Classifiers, Proc. Natl. Acad. Sci. USA 118 (2021), e2016191118. https://doi.org/10.1073/PNAS.2016191118.
  35. A. Swaminathan, U. Srivastava, L. Tu, I. Lopez, N.H. Shah, et al., Against Reflexive Recalibration: Towards a Causal Framework for Addressing Miscalibration, Diagn. Progn. Res. 9 (2025), 4. https://doi.org/10.1186/S41512-024-00184-2.
  36. L. Zhao, Y. Leng, Y. Hu, J. Xiao, Q. Li, et al., Understanding Decision Curve Analysis in Clinical Prediction Model Research, Postgrad. Med. J. 100 (2024), 512-515. https://doi.org/10.1093/POSTMJ/QGAE027.
  37. T. Honda, Y. Furuta, A. Maezono, S. Chen, Y. Ishida, et al., External Validation of a Risk Prediction Model for Atherosclerotic Cardiovascular Diseases in a Large National Health‐Checkup and Claim Database, J. Am. Hear. Assoc. 14 (2025), e040386. https://doi.org/10.1161/JAHA.124.040386.
  38. I. Hozo, G. Guyatt, B. Djulbegovic, Decision Curve Analysis Based on Summary Data, J. Eval. Clin. Pract. 30 (2023), 281-289. https://doi.org/10.1111/JEP.13945.
  39. G. Netto Flores Cruz, K. Korthauer, Bayesian Decision Curve Analysis with Bayesdca, Stat. Med. 43 (2024), 6042-6058. https://doi.org/10.1002/SIM.10277.
  40. C. Chai, S.Z. Peng, R. Zhang, C.W. Li, Y. Zhao, Advancing Emergency Department Triage Prediction with Machine Learning to Optimize Triage for Abdominal Pain Surgery Patients, Surg. Innov. 31 (2024), 583-597. https://doi.org/10.1177/15533506241273449.
  41. Q. Chen, Y. Qin, Z. Jin, X. Zhao, J. He, et al., Enhancing Performance of the National Field Triage Guidelines Using Machine Learning: Development of a Prehospital Triage Model to Predict Severe Trauma, J. Med. Internet Res. 26 (2024), e58740. https://doi.org/10.2196/58740.
  42. A.J. Vickers, B. Van Claster, L. Wynants, E.W. Steyerberg, Decision Curve Analysis: Confidence Intervals and Hypothesis Testing for Net Benefit, Diagn. Progn. Res. 7 (2023), 11. https://doi.org/10.1186/S41512-023-00148-Y.
  43. L. Rountree, Y.T. Lin, C. Liu, M. Salvatore, A. Admon, et al., Reporting of Fairness Metrics in Clinical Risk Prediction Models Used for Precision Health: Scoping Review, Online J. Public Health Inform. 17 (2025), e66598. https://doi.org/10.2196/66598.
  44. H.T. Cronjé, A. Katsiferis, L.K. Elsenburg, T.O. Andersen, N.H. Rod, et al., Assessing Racial Bias in Type 2 Diabetes Risk Prediction Algorithms, PLOS Glob. Public Health 3 (2023), e0001556. https://doi.org/10.1371/JOURNAL.PGPH.0001556.
  45. T. Hastie, R. Tibshirani, Generalized Additive Models, Stat. Sci. 1 (1986), 297–310. https://doi.org/10.1214/SS/1177013604.
  46. R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, et al., Intelligible Models for Healthcare: Predicting Pneumonia Risk and Hospital 30-Day Readmission, in: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, 2015, pp. 1721-1730. https://doi.org/10.1145/2783258.2788613.
  47. M.T. Ribeiro, S. Singh, C. Guestrin, "Why Should I Trust You?": Explaining the Predictions of Any Classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, 2016, pp. 1135-1144. https://doi.org/10.1145/2939672.2939778.
  48. S.M. Lundberg, G. Erion, H. Chen, A. DeGrave, J.M. Prutkin, et al., From Local Explanations to Global Understanding with Explainable AI for Trees, Nat. Mach. Intell. 2 (2020), 56-67. https://doi.org/10.1038/s42256-019-0138-9.
  49. S.M. Lundberg, S.I. Lee, A Unified Approach to Interpreting Model Predictions, in: 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017.
  50. J.H. Friedman, Greedy Function Approximation: A Gradient Boosting Machine, Ann. Stat. 29 (2001), 1189-1232. https://doi.org/10.1214/AOS/1013203451.
  51. D.W. Apley, J. Zhu, Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models, J. R. Stat. Soc. Ser. B: Stat. Methodol. 82 (2020), 1059-1086. https://doi.org/10.1111/RSSB.12377.
  52. S. Kaufman, S. Rosset, C. Perlich, O. Stitelman, Leakage in Data Mining, ACM Trans. Knowl. Discov. Data 6 (2012), 1-21. https://doi.org/10.1145/2382577.2382579.
  53. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, et al., Model Cards for Model Reporting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, ACM, New York, NY, USA, 2019, pp. 220-229. https://doi.org/10.1145/3287560.3287596.
  54. C. Rudin, Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, Nat. Mach. Intell. 1 (2019), 206-215. https://doi.org/10.1038/s42256-019-0048-x.
  55. J. Hippisley-Cox, C.A.C. Coupland, M. Bafadhel, R.E.K. Russell, A. Sheikh, et al., Development and Validation of a New Algorithm for Improved Cardiovascular Risk Prediction, Nat. Med. 30 (2024), 1440-1447. https://doi.org/10.1038/s41591-024-02905-y.
  56. M. Sud, A. Sivaswamy, A. Chu, P.C. Austin, T.J. Anderson, et al., Population-Based Recalibration of the Framingham Risk Score and Pooled Cohort Equations, J. Am. Coll. Cardiol. 80 (2022), 1330-1342. https://doi.org/10.1016/J.JACC.2022.07.026.
  57. K.R. van Daalen, D. Zhang, S. Kaptoge, E. Paige, E. Di Angelantonio, et al., Risk Estimation for the Primary Prevention of Cardiovascular Disease: Considerations for Appropriate Risk Prediction Model Selection, Lancet Glob. Health 12 (2024), e1343-e1358. https://doi.org/10.1016/S2214-109X(24)00210-9.