TY - JOUR
T1 - Machine learning approach to reduce alert fatigue using a disease medication–related clinical decision support system
T2 - Model development and validation
AU - Poly, Tahmina Nasrin
AU - Islam, Md Mohaimenul
AU - Muhtar, Muhammad Solihuddin
AU - Yang, Hsuan Chia
AU - Nguyen, Phung Anh
AU - Li, Yu Chuan
N1 - Funding Information:
We would like to thank AESOP (AI-Enhanced Safety of Prescription) Technology for giving us data and technological support to conduct this study. This research was funded, in part, by the Ministry of Education (MOE) (grant numbers MOE 109-6604-001-400 and DP2-109-21121-01-A-01) and the Ministry of Science and Technology (MOST) (grant number MOST109-2823-8-038-004).
Publisher Copyright:
©Tahmina Nasrin Poly, Md.Mohaimenul Islam, Muhammad Solihuddin Muhtar, Hsuan-Chia Yang, Phung Anh (Alex) Nguyen, Yu-Chuan (Jack) Li.
Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.
PY - 2020/11
Y1 - 2020/11
N2 - Background: Computerized physician order entry (CPOE) systems are incorporated into clinical decision support systems (CDSSs) to reduce medication errors and improve patient safety. Automatic alerts generated from CDSSs can directly assist physicians in making useful clinical decisions and can help shape prescribing behavior. Multiple studies reported that approximately 90%-96% of alerts are overridden by physicians, which raises questions about the effectiveness of CDSSs. There is intense interest in developing sophisticated methods to combat alert fatigue, but there is no consensus on the optimal approaches so far. Objective: Our objective was to develop machine learning prediction models to predict physicians’ responses in order to reduce alert fatigue from disease medication–related CDSSs. Methods: We collected data from a disease medication–related CDSS from a university teaching hospital in Taiwan. We considered prescriptions that triggered alerts in the CDSS between August 2018 and May 2019. Machine learning models, such as artificial neural network (ANN), random forest (RF), naïve Bayes (NB), gradient boosting (GB), and support vector machine (SVM), were used to develop prediction models. The data were randomly split into training (80%) and testing (20%) datasets. Results: A total of 6453 prescriptions were used in our model. The ANN machine learning prediction model demonstrated excellent discrimination (area under the receiver operating characteristic curve [AUROC] 0.94; accuracy 0.85), whereas the RF, NB, GB, and SVM models had AUROCs of 0.93, 0.91, 0.91, and 0.80, respectively. The sensitivity and specificity of the ANN model were 0.87 and 0.83, respectively. Conclusions: In this study, ANN showed substantially better performance in predicting individual physician responses to an alert from a disease medication–related CDSS, as compared to the other models. To our knowledge, this is the first study to use machine learning models to predict physician responses to alerts; furthermore, it can help to develop sophisticated CDSSs in real-world clinical settings.
AB - Background: Computerized physician order entry (CPOE) systems are incorporated into clinical decision support systems (CDSSs) to reduce medication errors and improve patient safety. Automatic alerts generated from CDSSs can directly assist physicians in making useful clinical decisions and can help shape prescribing behavior. Multiple studies reported that approximately 90%-96% of alerts are overridden by physicians, which raises questions about the effectiveness of CDSSs. There is intense interest in developing sophisticated methods to combat alert fatigue, but there is no consensus on the optimal approaches so far. Objective: Our objective was to develop machine learning prediction models to predict physicians’ responses in order to reduce alert fatigue from disease medication–related CDSSs. Methods: We collected data from a disease medication–related CDSS from a university teaching hospital in Taiwan. We considered prescriptions that triggered alerts in the CDSS between August 2018 and May 2019. Machine learning models, such as artificial neural network (ANN), random forest (RF), naïve Bayes (NB), gradient boosting (GB), and support vector machine (SVM), were used to develop prediction models. The data were randomly split into training (80%) and testing (20%) datasets. Results: A total of 6453 prescriptions were used in our model. The ANN machine learning prediction model demonstrated excellent discrimination (area under the receiver operating characteristic curve [AUROC] 0.94; accuracy 0.85), whereas the RF, NB, GB, and SVM models had AUROCs of 0.93, 0.91, 0.91, and 0.80, respectively. The sensitivity and specificity of the ANN model were 0.87 and 0.83, respectively. Conclusions: In this study, ANN showed substantially better performance in predicting individual physician responses to an alert from a disease medication–related CDSS, as compared to the other models. To our knowledge, this is the first study to use machine learning models to predict physician responses to alerts; furthermore, it can help to develop sophisticated CDSSs in real-world clinical settings.
KW - Alert fatigue
KW - Artificial neural network
KW - Clinical decision support system
KW - Machine learning
UR - http://www.scopus.com/inward/record.url?scp=85097472121&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85097472121&partnerID=8YFLogxK
U2 - 10.2196/19489
DO - 10.2196/19489
M3 - Article
AN - SCOPUS:85097472121
SN - 2291-9694
VL - 8
JO - JMIR medical informatics
JF - JMIR medical informatics
IS - 11
M1 - e19489
ER -