From Probabilities to Workload: Calibrated ML and Decision-Curve Analysis for Passenger-Service Operations

Article Sidebar

PDF
DOI: https://doi.org/10.28924/2291-8639-24-2026-250
Published: Aug 17, 2026
Cite as:
Thanyaporn Sukdet, Warawut Narkbunnum, From Probabilities to Workload: Calibrated ML and Decision-Curve Analysis for Passenger-Service Operations, Int. J. Anal. Appl., 24 (2026), 250.

Main Article Content

Thanyaporn Sukdet, Warawut Narkbunnum

Abstract

Airline passenger-service operations increasingly rely on predictive decision support to triage large volumes of service cases under capacity constraints. However, many studies emphasize ranking performance while providing limited evidence on the reliability of probabilities and the operational feasibility of threshold-based deployment. This study examines whether calibrated probabilities from machine learning (ML) models can support actionable, workload-aware decisions in airline passenger service operations. We implement a leakage-controlled pipeline with frozen preprocessing and compare regularized logistic regression, random forests, and gradient-boosting trees. Predicted probabilities are calibrated on non-test data using isotonic regression. Evaluation on a held-out test set integrates discrimination, probability-quality metrics, and calibration diagnostics. Decision curve analysis (DCA) quantifies net benefit relative to treat-none and treat-all strategies across an operational threshold window (0.05–0.15). To translate decision policies into operational implications, thresholds are mapped to workload units, including alerts per day and expected counts of true and false positives per day, assuming a nominal daily volume of 1,200 cases. Segment audits assess robustness across cabin class, travel type, delay band, and distance band. After calibration, the selected gradient-boosting model yields positive net benefit throughout the threshold window and forms the upper envelope relative to baseline strategies. Workload translation indicates that thresholds in the 0.05–0.15 range preserve true-positive throughput while reducing false-positive alerts, enabling managers to tune intervention capacity with minimal loss of decision value. Segment audits indicate generally stable performance, while identifying localized conditions in which modest segment-specific thresholds or light post-alert triage may reduce unwarranted escalations. Overall, calibrated ML, combined with DCA and workload translation, provides operationally feasible, governance-ready support for passenger-service decision making, improving reliability and efficiency while allowing threshold policies to adapt to daily capacity and evolving operating conditions.

Article Details

References

  1. M.H. Kim, J.W. Park, Y.J. Choi, A Study on the Effects of Waiting Time for Airport Security Screening Service on Passengers’ Emotional Responses and Airport Image, Sustainability 12 (2020), 10634. https://doi.org/10.3390/su122410634.
  2. ACI World, Joint ACI World-ICAO Passenger Traffic Report, Trends, and Outlook, Advisory Bulletin, Airports Council International, Montreal, 28 January 2025. https://aci.aero/2025/01/28/joint-aci-world-icao-passenger-traffic-report-trends-and-outlook/.
  3. K. Abdulaziz Alnowibet, A. Khireldin, M. Abdelawwad, A. Wagdy Mohamed, Airport Terminal Building Capacity Evaluation Using Queuing System, Alexandria Eng. J. 61 (2022), 10109–10118. https://doi.org/10.1016/j.aej.2022.03.055.
  4. Z.A. Marshall, J.H. Mott, A.J. Gottwald, C.A. Patrick, L.R.I. Dy, Expediting Airport Security Queues Through Advanced Lane Assignment, J. Transp. Secur. 15 (2022), 245–262. https://doi.org/10.1007/s12198-022-00247-9.
  5. EUROCONTROL, All-Causes Delays to Air Transport in Europe – Annual 2023, CODA Digest, EUROCONTROL, 16 December 2024. https://www.eurocontrol.int/publication/all-causes-delays-air-transport-europe-annual-2023.
  6. P. Fleurquin, J.J. Ramasco, V.M. Eguiluz, Systemic Delay Propagation in the US Airport Network, Sci. Rep. 3 (2013), 1159. https://doi.org/10.1038/srep01159.
  7. A. Brun, E. Feron, S. Alam, D. Delahaye, Schedule Optimization and Staff Allocation for Airport Security Checkpoints Using Guided Simulated Annealing and Integer Linear Programming, J. Air Transp. Manag. 124 (2025), 102746. https://doi.org/10.1016/j.jairtraman.2025.102746.
  8. M. Dai, A Hybrid Machine Learning-Based Model for Predicting Flight Delay through Aviation Big Data, Sci. Rep. 14 (2024), 4603. https://doi.org/10.1038/s41598-024-55217-z.
  9. Z. Guo, B. Yu, M. Hao, W. Wang, Y. Jiang, F. Zong, A Novel Hybrid Method for Flight Departure Delay Prediction Using Random Forest Regression and Maximal Information Coefficient, Aerosp. Sci. Technol. 116 (2021), 106822. https://doi.org/10.1016/j.ast.2021.106822.
  10. D. Bala Bisandu, I. Moulitsas, Prediction of Flight Delay Using Deep Operator Network with Gradient-Mayfly Optimisation Algorithm, Expert Syst. Appl. 247 (2024), 123306. https://doi.org/10.1016/j.eswa.2024.123306.
  11. R. Murugesan, R. A P, N. N, R. Balanathan, Forecasting Airline Passengers’ Satisfaction Based on Sentiments and Ratings: An Application of VADER and Machine Learning Techniques, J. Air Transp. Manag. 120 (2024), 102668. https://doi.org/10.1016/j.jairtraman.2024.102668.
  12. H. Mirzahossein, S. Rezashoar, Feature Importance Analysis of Optimized Machine Learning Modeling for Predicting Customers Satisfaction at the United States Airlines, Mach. Learn. Appl. 22 (2025), 100734. https://doi.org/10.1016/j.mlwa.2025.100734.
  13. S. Moosavi, R. Ramnath, Context-Aware Driver Risk Prediction with Telematics Data, Accid. Anal. Prev. 192 (2023), 107269. https://doi.org/10.1016/j.aap.2023.107269.
  14. J. Boylan, D. Meyer, W.S. Chen, A Systematic Review of the Use of In-Vehicle Telematics in Monitoring Driving Behaviours, Accid. Anal. Prev. 199 (2024), 107519. https://doi.org/10.1016/j.aap.2024.107519.
  15. H. Mao, F. Guo, X. Deng, Z.R. Doerzaph, Decision-Adjusted Driver Risk Predictive Models Using Kinematics Information, Accid. Anal. Prev. 156 (2021), 106088. https://doi.org/10.1016/j.aap.2021.106088.
  16. T. Silva Filho, H. Song, M. Perello-Nieto, R. Santos-Rodriguez, M. Kull, P. Flach, Classifier Calibration: A Survey on How to Assess and Improve Predicted Class Probabilities, Mach. Learn. 112 (2023), 3211–3260. https://doi.org/10.1007/s10994-023-06336-7.
  17. M. Börjesson, I. Rubensson, Satisfaction with Crowding and Other Attributes in Public Transport, Transp. Policy 79 (2019), 213–222. https://doi.org/10.1016/j.tranpol.2019.05.010.
  18. J. Soza-Parra, S. Raveau, J.C. Muñoz, O. Cats, The Underlying Effect of Public Transport Reliability on Users’ Satisfaction, Transp. Res. Part A Policy Pract. 126 (2019), 83–93. https://doi.org/10.1016/j.tra.2019.06.004.
  19. M. Alfarhood, R. Alotaibi, B. Abdulrahim, A. Einieh, M. Almousa, A. Alkhanifer, Predicting Flight Delays with Machine Learning: A Case Study from Saudi Arabian Airlines, Int. J. Aerosp. Eng. 2024 (2024), 3385463. https://doi.org/10.1155/2024/3385463.
  20. A.J. Vickers, E.B. Elkin, Decision Curve Analysis: A Novel Method for Evaluating Prediction Models, Med. Decis. Mak. 26 (2006), 565–574. https://doi.org/10.1177/0272989X06295361.
  21. A.J. Vickers, B. van Calster, E.W. Steyerberg, A Simple, Step-by-Step Guide to Interpreting Decision Curve Analysis, Diagn. Progn. Res. 3 (2019), 18. https://doi.org/10.1186/s41512-019-0064-7.
  22. M. Hardt, E. Price, N. Srebro, Equality of Opportunity in Supervised Learning, in: Advances in Neural Information Processing Systems 29 (NIPS 2016), Curran Associates, 2016, pp. 3315–3323. https://doi.org/10.48550/arXiv.1610.02413.
  23. X. Zhang, Q. Ke, X. Zhao, Travel Demand Forecasting: A Fair AI Approach, IEEE Trans. Intell. Transp. Syst. 25 (2024), 14611–14627. https://doi.org/10.1109/TITS.2024.3395061.
  24. Y. Zheng, S. Wang, J. Zhao, Equality of Opportunity in Travel Behavior Prediction with Deep Neural Networks and Discrete Choice Models, Transp. Res. Part C Emerg. Technol. 132 (2021), 103410. https://doi.org/10.1016/j.trc.2021.103410.
  25. G.W. Brier, Verification of Forecasts Expressed in Terms of Probability, Mon. Weather Rev. 78 (1950), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.
  26. D.B. Stephenson, C.A.S. Coelho, I.T. Jolliffe, Two Extra Components in the Brier Score Decomposition, Weather Forecast. 23 (2008), 752–757. https://doi.org/10.1175/2007WAF2006116.1.
  27. S. Siegert, Simplifying and Generalising Murphy’s Brier Score Decomposition, Q. J. R. Meteorol. Soc. 143 (2017), 1178–1183. https://doi.org/10.1002/qj.2985.
  28. A.H. Murphy, A New Vector Partition of the Probability Score, J. Appl. Meteorol. 12 (1973), 595–600. https://doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.
  29. C. Guo, G. Pleiss, Y. Sun, K.Q. Weinberger, On Calibration of Modern Neural Networks, in: Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  30. A. Niculescu-Mizil, R. Caruana, Predicting Good Probabilities with Supervised Learning, in: Proceedings of the 22nd International Conference on Machine Learning – ICML ’05, ACM Press, New York, 2005, pp. 625–632. https://doi.org/10.1145/1102351.1102430.
  31. G. Gao, S. Meng, M.V. Wüthrich, What Can We Learn from Telematics Car Driving Data: A Survey, Insur. Math. Econ. 104 (2022), 185–199. https://doi.org/10.1016/j.insmatheco.2022.02.004.
  32. M.R. Islam, D. Wang, M. Abdel-Aty, Calibrated Confidence Learning for Large-Scale Real-Time Crash and Severity Prediction, npj Sustain. Mobil. Transp. 1 (2024), 1. https://doi.org/10.1038/s44333-024-00001-9.
  33. J. Wei, Y. Miyazaki, F. Sato, Pre-Crash Injury Risk Prediction with Guaranteed Confidence Level: A Conformal and Interpretable Framework, Traffic Inj. Prev. 27 (2026), 583–593. https://doi.org/10.1080/15389588.2025.2538725.
  34. M. Sarhani, A. Nourmohammadzadeh, S. Voß, M. El Amrani, Predicting and Analyzing Ferry Transit Delays Using Open Data and Machine Learning, J. Public Transp. 27 (2025), 100124. https://doi.org/10.1016/j.jpubtr.2025.100124.
  35. E. Ruiz, W.F. Yushimito, L. Aburto, R. de la Cruz, Predicting Passenger Satisfaction in Public Transportation Using Machine Learning Models, Transp. Res. Part A Policy Pract. 181 (2024), 103995. https://doi.org/10.1016/j.tra.2024.103995.
  36. F. Rocco di Torrepadula, E.V. Napolitano, S. Di Martino, N. Mazzocca, Machine Learning for Public Transportation Demand Prediction: A Systematic Literature Review, Eng. Appl. Artif. Intell. 137 (2024), 109166. https://doi.org/10.1016/j.engappai.2024.109166.
  37. M. Attioui, M. Lahby, Congestion Forecasting Using Machine Learning Techniques: A Systematic Review, Future Transp. 5 (2025), 76. https://doi.org/10.3390/futuretransp5030076.
  38. J. Liang, J. Ke, S. Feng, H. Yang, Machine Learning for On-Demand Ride Services: A Survey, Artif. Intell. Transp. 2 (2025), 100008. https://doi.org/10.1016/j.ait.2025.100008.
  39. T. Peng, J. Gao, O. Cats, Uncertainty-Aware Probabilistic Travel Demand Prediction for Mobility-on-Demand Services, Transp. Res. Part C Emerg. Technol. 181 (2025), 105383. https://doi.org/10.1016/j.trc.2025.105383.
  40. J. Yang, L. Chen, Z. Su, W. Ma, Z. Zou, K. An, Decision-Focused Learning for Optimal Subsidy Allocation in Ride-Hailing Services, Transp. Res. Part C Emerg. Technol. 180 (2025), 105301. https://doi.org/10.1016/j.trc.2025.105301.
  41. Y. Li, R. Yu, C. Shahabi, Y. Liu, Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting, in: 6th International Conference on Learning Representations (ICLR 2018), 2018. https://doi.org/10.48550/arXiv.1707.01926.
  42. B. Yu, H. Yin, Z. Zhu, Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting, in: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI, California, 2018, pp. 3634–3640. https://doi.org/10.24963/ijcai.2018/505.
  43. T. Gneiting, A.E. Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, J. Am. Stat. Assoc. 102 (2007), 359–378. https://doi.org/10.1198/016214506000001437.
  44. J. Błasiok, P. Nakkiran, Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing, in: 12th International Conference on Learning Representations (ICLR 2024), 2024. https://doi.org/10.48550/arXiv.2309.12236.
  45. Z. Jiang, A. Liu, B. Van Durme, Addressing the Binning Problem in Calibration Assessment through Scalar Annotations, Trans. Assoc. Comput. Linguist. 12 (2024), 120–136. https://doi.org/10.1162/tacl_a_00636.
  46. G.S. Collins, P. Dhiman, J. Ma, M.M. Schlussel, L. Archer, B. Van Calster, F.E. Harrell, G.P. Martin, K.G.M. Moons, M. van Smeden, M. Sperrin, G.S. Bullock, R.D. Riley, Evaluation of Clinical Prediction Models (Part 1): From Development to External Validation, BMJ 384 (2024), e074819. https://doi.org/10.1136/bmj-2023-074819.
  47. B. Van Calster, E.W. Steyerberg, L. Wynants, M. van Smeden, There Is No Such Thing as a Validated Prediction Model, BMC Med. 21 (2023), 70. https://doi.org/10.1186/s12916-023-02779-w.
  48. J. Bröcker, L.A. Smith, Increasing the Reliability of Reliability Diagrams, Weather Forecast. 22 (2007), 651–661. https://doi.org/10.1175/WAF993.1.
  49. A.N. Angelopoulos, S. Bates, Conformal Prediction: A Gentle Introduction, Found. Trends Mach. Learn. 16 (2023), 494–591. https://doi.org/10.1561/2200000101.
  50. A.J. Vickers, B. Van Claster, L. Wynants, E.W. Steyerberg, Decision Curve Analysis: Confidence Intervals and Hypothesis Testing for Net Benefit, Diagn. Progn. Res. 7 (2023), 11. https://doi.org/10.1186/s41512-023-00148-y.
  51. G.N.F. Cruz, K. Korthauer, Bayesian Decision Curve Analysis with bayesDCA, Stat. Med. 43 (2024), 6042–6058. https://doi.org/10.1002/sim.10277.
  52. X. Chen, S. Bai, Y. Wei, H. Jiang, How Income Satisfaction Impacts Driver Engagement Dynamics in Ride-Hailing Services, Transp. Res. Part C Emerg. Technol. 157 (2023), 104418. https://doi.org/10.1016/j.trc.2023.104418.
  53. N. Xie, W. Tang, J. Zhu, J. Li, X. Chen, Understanding Causal Effects of Ride-Sourcing Subsidy: A Novel Generative Adversarial Networks Approach, Transp. Res. Part C Emerg. Technol. 157 (2023), 104371. https://doi.org/10.1016/j.trc.2023.104371.
  54. P. Ashkrof, G.H. de Almeida Correia, O. Cats, B. van Arem, On the Relocation Behavior of Ride-Sourcing Drivers, Transp. Lett. 16 (2024), 330–337. https://doi.org/10.1080/19427867.2023.2192581.
  55. J.G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N.V. Chawla, F. Herrera, A Unifying View on Dataset Shift in Classification, Pattern Recognit. 45 (2012), 521–530. https://doi.org/10.1016/j.patcog.2011.06.019.
  56. J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, A. Bouchachia, A Survey on Concept Drift Adaptation, ACM Comput. Surv. 46 (2014), 1–37. https://doi.org/10.1145/2523813.
  57. Q. Hou, Y. Yang, J. Liang, X. Huo, J. Leng, A Deep Transfer Learning Approach for Real-Time Traffic Conflict Prediction with Trajectory Data, Accid. Anal. Prev. 214 (2025), 107966. https://doi.org/10.1016/j.aap.2025.107966.
  58. X. Zou, E. Chung, Traffic Prediction via Clustering and Deep Transfer Learning with Limited Data, Comput.-Aided Civ. Infrastruct. Eng. 39 (2024), 2683–2700. https://doi.org/10.1111/mice.13207.
  59. J. Shao, S. Li, K. Zhang, A. Wang, M. Li, Cross-City Traffic Prediction Based on Deep Domain Adaptive Transfer Learning, Transp. Res. Part C Emerg. Technol. 176 (2025), 105152. https://doi.org/10.1016/j.trc.2025.105152.
  60. E.W. Steyerberg, Y. Vergouwe, Towards Better Clinical Prediction Models: Seven Steps for Development and an ABCD for Validation, Eur. Heart J. 35 (2014), 1925–1931. https://doi.org/10.1093/eurheartj/ehu207.
  61. J. Xu, Y. Xiao, W.H. Wang, Y. Ning, E.A. Shenkman, J. Bian, F. Wang, Algorithmic Fairness in Computational Medicine, EBioMedicine 84 (2022), 104250. https://doi.org/10.1016/j.ebiom.2022.104250.
  62. B. Bharti, P. Yi, J. Sulam, Estimating and Controlling for Equalized Odds via Sensitive Attribute Predictors, Adv. Neural Inf. Process. Syst. 36 (2023), 37173–37192. https://pmc.ncbi.nlm.nih.gov/articles/PMC11167624/.
  63. E. Small, K. Sokol, D. Manning, F.D. Salim, J. Chan, Equalised Odds Is Not Equal Individual Odds: Post-Processing for Group and Individual Fairness, in: The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24), ACM, New York, 2024, pp. 1559–1578. https://doi.org/10.1145/3630106.3658989.
  64. Y. Liang, Fairness-Aware Dynamic Ride-Hailing Matching Based on Reinforcement Learning, Electronics 13 (2024), 775. https://doi.org/10.3390/electronics13040775.
  65. T.W. Sanchez, Y. Qian, X. Yan, AI Applications in Transportation and Equity: A Survey of U.S. Transportation Professionals, Future Transp. 4 (2024), 1161–1176. https://doi.org/10.3390/futuretransp4040056.
  66. B.C. Cheong, Transparency and Accountability in AI Systems: Safeguarding Wellbeing in the Age of Algorithmic Decision-Making, Front. Hum. Dyn. 6 (2024), 1421273. https://doi.org/10.3389/fhumd.2024.1421273.
  67. K. Hamad, E. Alotaibi, W. Zeiada, G. Al-Khateeb, S. Abu Dabous, M. Omar, B.R.K. Mantha, M.G. Arab, T. Merabtene, Explainable Artificial Intelligence Visions on Incident Duration Using eXtreme Gradient Boosting and SHapley Additive exPlanations, Multimodal Transp. 4 (2025), 100209. https://doi.org/10.1016/j.multra.2025.100209.
  68. L. Obaid, K. Hamad, S. Barakat, Incident Duration Reliability Assessment Using Monte-Carlo Simulation and Kernel Density Estimation of Machine Learning-Based Models, Int. J. Transp. Sci. Technol. 20 (2025), 157–177. https://doi.org/10.1016/j.ijtst.2024.11.005.
  69. C. Oprea, M. Rosca, E. Rosca, I. Costea, A. Ilie, O. Dinu, A. Ruscă, Analyzing Passenger Flows in an Airport Terminal: A Discrete Simulation Model, Computation 12 (2024), 223. https://doi.org/10.3390/computation12110223.
  70. O. Efthimiou, M. Seo, K. Chalkou, T. Debray, M. Egger, G. Salanti, Developing Clinical Prediction Models: A Step-by-Step Guide, BMJ 386 (2024), e078276. https://doi.org/10.1136/bmj-2023-078276.
  71. T. Dong, S. Sinha, B. Zhai, D. Fudulu, J. Chan, P. Narayan, A. Judge, M. Caputo, A. Dimagli, U. Benedetto, G.D. Angelini, Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis, JMIRx Med 5 (2024), e45973. https://doi.org/10.2196/45973.
  72. F. Hinder, V. Vaquet, B. Hammer, One or Two Things We Know About Concept Drift—A Survey on Monitoring in Evolving Environments. Part A: Detecting Concept Drift, Front. Artif. Intell. 7 (2024), 1330257. https://doi.org/10.3389/frai.2024.1330257.