|Table of Contents|

Evaluation of drivers’ mental workload based on multi-modal physiological signals(PDF)

Journal of Shenzhen University Science and Engineering[ISSN:1000-2618/CN:44-1401/N]

Issue:
2022 Vol.39 No.3(237-362)
Page:
278-286
Research Field:
Transportation Logistics

Info

Title:
Evaluation of drivers’ mental workload based on multi-modal physiological signals
Author(s):
ZHANG Qiliang;  YANG Kunhua;  QU Xingda;  and TAO Da
Institute of Human Factors and Ergonomics, Shenzhen University, Shenzhen 518060, Guangdong Province, P. R. China
Keywords:
traffic information engineering and control;  mental workload;  machine learning;  multi-modal;  electroencephalographic signal;  electrocardiographic signal;  electrodermal activity signal
PACS:
U491；TN911.72
DOI:
10.3724/SP.J.1249.2022.03278
Abstract:
Accurately assessing the driver’s mental workload is of great significance to reduce the traffic accidents caused by the driver’s mental overload. This study aims to evaluate drivers’ mental workload in simulated typical driving scenarios, with N-back cognitive tasks used to manipulate varied levels of task difficulty. We collect data on multi-modal physiological signals including electroencephalogram (EEG), electrocardiogram (ECG), and electrodermal activity (EDA) signals, and subjective mental load of the National Aeronautics and Space Administration task load index (NASA_TLX) during the task completion process of the driver in the experiment, and propose a series of mental workload classification models based on feature analysis and pattern recognition of the multi-modal physiological signals. These classification models are verified by machine learning algorithms of random forest, decision tree and k-nearest neighbor models. The results show that the accuracy of classification models varies with different modalities of physiological signals. EEG-based classification models yield the highest accuracy among single-modal classification models, followed by EDA-based and ECG-based models. Multi-modal-based classification models generally perform better than single-modal classification models. The random forest classification algorithm based on three-modal physiological signals of EEG, ECG and EDA has the highest classification accuracy.

References:

［1］ 郑玲，乔旭强，倪涛，等. 基于多维信息特征分析的驾驶人认知负荷研究［J］. 中国公路学报，2021，34（4）：240-250. ZHENG Ling, QIAO Xuqiang, NI Tao, et al. Driver congnitive loads based on multi-dimensional information feature analysis [J]. China Journal of Highway and Transport, 2021, 34(4): 240-250. (in Chinese) ［2］ National Safety Council. Understanding the distracted brain: why driving while using hands-free cell phones is risky behavior [R]. Spring Lake: National Safety Council, 2012. ［3］ GALY E. Consideration of several mental workload categories: perspectives for elaboration of new ergonomic recommendations concerning shiftwork [J]. Theoretical Issues in Ergonomics Science, 2018, 19(4): 483-497. ［4］ DONG Yanchao, HU Zhencheng, UCHIMURA K, et al. Driver inattention monitoring system for intelligent vehicles: a review [J]. IEEE Transactions on Intelligent Transportation Systems, 2010, 12(2): 596-614. ［5］ DI FLUMERI G, BORGHINI G, ARIC? P, et al. EEG-based mental workload neurometric to evaluate the impact of different traffic and road conditions in real driving settings [J]. Frontiers in Human Neuroscience, 2018, 12: 509. ［6］ JIMENEZ-MOLINA A, RETAMAL C, LIRA H. Using psychophysiological sensors to assess mental workload in web browsing [J]. Sensors, 2017, 18(2): 458. ［7］ OMURTAG A, ROY R N, DEHAIS F, et al. Tracking team mental workload by multimodal measurements in the operating room [M/OL]. Neuroergonomics-The Brain at Work and in Everyday Life. New York, USA: 2019: 99-103. ［8］ KOSTI M A, GEORGIADIS K, ADAMOS D A, et al. Towards an affordable brain computer interface for the assessment of programmers’ mental workload [J]. International Journal of Human Computer Studies, 2018, 115: 52-66. ［9］ DIAZ-PIEDRA C, SEBASTI?N M V, DI STASI L L. EEG theta power activity reflects workload among army combat drivers: an experimental study [J]. Brain Sciences, 2020, 10(4): 199. ［10］ DE RIVECOURT M, KUPERUS M, POST W, et al. Cardiovascular and eye activity measures as indices for momentary changes in mental effort during simulated flight [J]. Ergonomics, 2008, 51(9): 1295-1319. ［11］ TAO Da, TAN Haibo, WANG Hailiang, et al. A systematic review of physiological measures of mental workload [J]. International Journal of Environmental Research and Public Health, 2019, 16(15): 2716. ［12］ HWANG S L， YAU Y J， LIN Y T， et al. Predicting work performance in nuclear power plants ［J］. Safety Science， 2008， 46（7）： 1115-1124. ［13］ FOY H J, CHAPMAN P J A E. Mental workload is reflected in driver behaviour, physiology, eye movements and prefrontal cortex activation [J]. Applied Ergonomics, 2018, 73: 90-99. ［14］ DING Yi, CAO Yaqin, DUFFY V G, et al. Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning [J]. Ergonomics, 2020, 63(7): 896-908. ［15］ DEBIE E, ROJAS R F, FIDOCK J, et al. Multimodal fusion for objective assessment of cognitive workload: a review [J]. IEEE Transactions on Cybernetics, 2019, 51(3): 1542-1555. ［16］ CINAZ B, ARNRICH B, LA MARCA R, et al. Monitoring of mental workload levels during an everyday life office-work scenario [J]. Personal Ubiquitous Computing, 2013, 17(2): 229-239. ［17］ MATTHEWS G, REINERMAN-JONES L E, BARBER D J, et al. The psychometrics of mental workload: multiple measures are sensitive but divergent [J]. Human Factors, 2015, 57(1): 125-143. ［18］ ZHAO Guozhen, LIU Yongjin, SHI Yuanchun. Real-time assessment of the cross-task mental workload using physiological measures during anomaly detection [J]. IEEE Transactions on Human-Machine Systems, 2018, 48(2): 149-160. ［19］ HOLZINGER A, KIESEBERG P, WEIPPL E, et al. Current advances, trends and challenges of machine learning and knowledge extraction: from machine learning to explainable AI [C]// International Cross-Domain Conference for Machine Learning and Knowledge Extraction. Hamburg, Germany: Springer, 2018: 1-8. ［20］ KOENIG A, REHG T, RASSHOFER R. Statistical sensor fusion of ECG data using automotive-grade sensors [J]. Advances in Radio Science, 2015, 13: 197-202. ［21］ YIN Zhong, ZHANG Jianhua. Operator functional state classification using least-square support vector machine based recursive feature elimination technique [J]. Computer Methods & Programs in Biomedicine, 2014, 113(1): 101-115. ［22］ YAN Shengyuan, TRAN C C, WEI Yingying, et al. Driver’s mental workload prediction model based on physiological indices [J]. International Journal of Occupational Safety & Ergonomics, 2017, 25(2): 1-37. ［23］ DAMOS D. Multiple-task performance [M]. Hants, UK: Taylor and Francis, 1991. ［24］ HEINE T, LENIS G, REICHENSPERGER P, et al. Electrocardiographic features for the measurement of drivers’ mental workload [J]. Applied Ergonomics, 2017, 61: 31-43. ［25］ FENG Chuanyan, WANYAN Xiaoru, YANG Kun, et al. A comprehensive prediction and evaluation method of pilot workload [J]. Technology and Health Care, 2018, 26(Suppl.1): 65-78. ［26］ BIAU G, SCORNET E. A random forest guided tour [J]. Test, 2016, 25(2): 197-227. ［27］ SO W K, WONG S W, MAK J N, et al. An evaluation of mental workload with frontal EEG [J]. PloS One, 2017, 12(4): e0174949. ［28］ ZHANG Minling, ZHOU Zhihua. ML-KNN: a lazy learning approach to multi-label learning [J]. Pattern Recognition, 2007, 40(7): 2038-2048. ［29］ FAN Xiaoli, ZHAO Chaoyi, ZHANG Xin, et al. Assessment of mental workload based on multi-physiological signals [J]. Technology Health Care, 2020, 28(Suppl.1): 67-80. ［30］ ISLAM M R, BARUA S, AHMED M U, et al. A novel mutual information based feature set for drivers’ mental workload evaluation using machine learning [J]. Brain Sciences, 2020, 10(8): 551.

Memo

Memo:
-