TY - JOUR
T1 - Emotion recognition in doctor-patient interactions from real-world clinical video database
T2 - Initial development of artificial empathy
AU - Huang, Chih Wei
AU - Wu, Bethany C.Y.
AU - Nguyen, Phung Anh
AU - Wang, Hsiao Han
AU - Kao, Chih Chung
AU - Lee, Pei Chen
AU - Rahmanti, Annisa Ristya
AU - Hsu, Jason C.
AU - Yang, Hsuan Chia
AU - Li, Yu Chuan Jack
N1 - Funding Information:
The authors would like to acknowledge the staff and participants in the dermatology outpatient clinic, Taipei Municipal Wanfang Hospital, and Taipei Medical University Hospital for their support. The authors would also like to thank Industrial Technology Research Institute (ITRI) for providing facial expression technology. This study was funded by the Ministry of Science and Technology (Grant no. MOST 110–2221-E-038–002-MY2), iGuardian project (NSTC 111–2622–8–038 −006 -IE), WFH cancer prediction project (NSTC 111–2321-B-038 −004), and the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan (Grant no. DP2–111–21121–01-A-02). The funder had no role in the preparation, review, or approval of the manuscript and decision to submit the manuscript for publication.
Funding Information:
This study was funded by the Ministry of Science and Technology (Grant no. MOST 110–2221-E-038–002-MY2 ), iGuardian project ( NSTC 111–2622–8–038 −006 -IE ), WFH cancer prediction project ( NSTC 111–2321-B-038 −004 ), and the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan (Grant no. DP2–111–21121–01-A-02 ). The funder had no role in the preparation, review, or approval of the manuscript and decision to submit the manuscript for publication.
Publisher Copyright:
© 2023
PY - 2023/5
Y1 - 2023/5
N2 - Background and objective: The promising use of artificial intelligence (AI) to emulate human empathy may help a physician engage with a more empathic doctor-patient relationship. This study demonstrates the application of artificial empathy based on facial emotion recognition to evaluate doctor-patient relationships in clinical practice. Methods: A prospective study used recorded video data of doctor-patient clinical encounters in dermatology outpatient clinics, Taipei Municipal Wanfang Hospital, and Taipei Medical University Hospital collected from March to December 2019. Two cameras recorded the facial expressions of four doctors and 348 adult patients during regular clinical practice. Facial emotion recognition was used to analyze the basic emotions of doctors and patients with a temporal resolution of 1 second. In addition, a physician-patient satisfaction questionnaire was administered after each clinical session, and two standard patients gave impartial feedback to avoid bias. Results: Data from 326 clinical session videos showed that (1) Doctors expressed more emotions than patients (t [326] > = 2.998, p < = 0.003), including anger, happiness, disgust, and sadness; the only emotion that patients showed more than doctors was surprise (t [326] = -4.428, p < .001) (p < .001). (2) Patients felt happier during the latter half of the session (t [326] = -2.860, p = .005), indicating a good doctor-patient relationship. Conclusions: Artificial empathy can offer objective observations on how doctors' and patients' emotions change. With the ability to detect emotions in 3/4 view and profile images, artificial empathy could be an accessible evaluation tool to study doctor-patient relationships in practical clinical settings.
AB - Background and objective: The promising use of artificial intelligence (AI) to emulate human empathy may help a physician engage with a more empathic doctor-patient relationship. This study demonstrates the application of artificial empathy based on facial emotion recognition to evaluate doctor-patient relationships in clinical practice. Methods: A prospective study used recorded video data of doctor-patient clinical encounters in dermatology outpatient clinics, Taipei Municipal Wanfang Hospital, and Taipei Medical University Hospital collected from March to December 2019. Two cameras recorded the facial expressions of four doctors and 348 adult patients during regular clinical practice. Facial emotion recognition was used to analyze the basic emotions of doctors and patients with a temporal resolution of 1 second. In addition, a physician-patient satisfaction questionnaire was administered after each clinical session, and two standard patients gave impartial feedback to avoid bias. Results: Data from 326 clinical session videos showed that (1) Doctors expressed more emotions than patients (t [326] > = 2.998, p < = 0.003), including anger, happiness, disgust, and sadness; the only emotion that patients showed more than doctors was surprise (t [326] = -4.428, p < .001) (p < .001). (2) Patients felt happier during the latter half of the session (t [326] = -2.860, p = .005), indicating a good doctor-patient relationship. Conclusions: Artificial empathy can offer objective observations on how doctors' and patients' emotions change. With the ability to detect emotions in 3/4 view and profile images, artificial empathy could be an accessible evaluation tool to study doctor-patient relationships in practical clinical settings.
KW - Artificial empathy
KW - Artificial intelligence
KW - Doctor-patient relations
KW - Emotion recognition
KW - Empathy
UR - http://www.scopus.com/inward/record.url?scp=85150830973&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85150830973&partnerID=8YFLogxK
U2 - 10.1016/j.cmpb.2023.107480
DO - 10.1016/j.cmpb.2023.107480
M3 - Article
C2 - 36965299
AN - SCOPUS:85150830973
SN - 0169-2607
VL - 233
JO - Computer Methods and Programs in Biomedicine
JF - Computer Methods and Programs in Biomedicine
M1 - 107480
ER -