[1]夏林中,叶剑锋,罗德安,等.基于BERT-BiLSTM模型的短文本自动评分系统[J].深圳大学学报理工版,2022,39(3):349-354.[doi:10.3724/SP.J.1249.2022.03349]
　XIA Linzhong,YE Jianfeng,LUO Dean,et al.Short text automatic scoring system based on BERT-BiLSTM model[J].Journal of Shenzhen University Science and Engineering,2022,39(3):349-354.[doi:10.3724/SP.J.1249.2022.03349]
点击复制

基于BERT-BiLSTM模型的短文本自动评分系统()
分享到：

《深圳大学学报理工版》[ISSN:1000-2618/CN:44-1401/N]

卷:
第39卷
期数:
2022年第3期
页码:
349-354
栏目:
电子与信息科学
出版日期:
2022-05-16

文章信息/Info

Title:
Short text automatic scoring system based on BERT-BiLSTM model
文章编号:
202203014
作者:
夏林中;  叶剑锋;  罗德安;  管明祥;  刘俊;  曹雪梅
深圳信息职业技术学院人工智能技术应用工程实验室，广东深圳 518172
Author(s):
XIA Linzhong;  YE Jianfeng;  LUO De’an;  GUAN Mingxiang;  LIU Jun;  and CAO Xuemei
Engineering Applications of Artificial Intelligence Technology Laboratory, Shenzhen Institute of Information Technology, Shenzhen 518172, Guangdong Province, P. R. China
关键词:
信号与信息处理; 自然语言处理; BERT语言模型; 短文本自动评分; 长短时记忆网络; 二次加权kappa系数
Keywords:
signal and information processing;  natural language processing;  BERT language model;  short text automatic scoring;  long short-term memory net;  quadratic weighted kappa coefficient
分类号:
TP18；H08
DOI:
10.3724/SP.J.1249.2022.03349
文献标志码:
A
摘要:
针对短文本自动评分中存在的特征稀疏、一词多义及上下文关联信息少等问题，提出一种基于BERT-BiLSTM（bidirectional encoder representations from transformers - bidirectional long short-term memory）的短文本自动评分模型．使用BERT（bidirectional encoder representations from transformers）语言模型预训练大规模语料库习得通用语言的语义特征，通过预训练好的BERT语言模型预微调下游具体任务的短文本数据集习得短文本的语义特征和关键词特定含义，再通过BiLSTM（bidirectional long short-term memory）捕获深层次上下文关联信息，最后将获得的特征向量输入Softmax回归模型进行自动评分．实验结果表明，对比CNN（convolutional neural networks）、CharCNN（character-level CNN）、LSTM（long short-term memory）和BERT等基准模型，基于BERT-BiLSTM的短文本自动评分模型所获的二次加权kappa系数平均值最优．
Abstract:
Aiming at the problems of sparse features, polysemy of one word and less context related information in short text automatic scoring, a short text automatic scoring model based on bidirectional encoder representations from transformers - bidirectional long short-term memory (BERT-BiLSTM) is proposed. Firstly, the large-scale corpus is pre-trained with bidirectional encoder representations from transformers (BERT) language model to acquire the semantic features of the general language. Then the semantic features of short text and the semantics of keywords in a specific context are acquired through the short text data for the pre-fine tuning downstream specific tasks set pre-fined by BERT. And then the deep-seated context dependency is captured through bidirectional long short-term memory (BiLSTM). Finally, the obtained feature vectors are input into Softmax regression model for automatic scoring. The experimental results show that compared with other benchmark models of convolutional neural networks(CNN), character-level CNN (CharCNN), long short-term memory (LSTM) and BERT, the short text automatic scoring model based on BERT-BiLSTM achieves the best average value of quadratic weighted kappa coefficient.

参考文献/References:

［1］ DIKLI S. An overview of automated scoring of essays [J]. Journal of Technology, Learning, and Assessment, 2006, 5(1): 1-35.
［2］ PAGE E B. The imminence of grading essays by computer [J]. Phi Delta Kappan, 1966, 48: 238-243.
［3］ CLAUDIA L, MARTIN C. C-rater: automated scoring of short-answer questions [J]. Computers and the Humanities, 2003, 37(4): 389-405.
［4］ DEERWESTER S, DUMAIS S T, FURNAS G W, et al. Indexing by latent semantic analysis [J]. Journal of the American Society for Information Science, 1990, 41(6): 391-407.
［5］ BLEI D M, NG A Y, JORDAN M I. Latent Dirichlet allocation [J]. Journal of Machine Learning Research, 2003, 3(4/5): 993-1022.
［6］ TANG D. Sentiment-specific representation learning for document-level sentiment analysis [C] // Proceedings of the 8th International Conference on Web Search and Data Mining. Shanghai, China: ACM, 2015: 447-452.
［7］ PANG B, LEE L. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales [C] // Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. Ann Arbor, USA: ACL, 2005: 115-124.
［8］ LEE K, HAN S, MYAENG S H. A discourse-aware neural network-based text model for document-level text classification [J]. Journal of Information Science, 2018, 44(6): 715-735.
［9］ MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space [EB/OL]. (2013- 09- 07) [2021- 08- 05]. https: //arxiv.org/abs/1301.3781.
［10］ COLLOBERT R, WESTON J, BOTTOU L, et al. Natural language processing (almost) from scratch [J]. Journal of Machine and Learning Research, 2011, 12: 2493-2537.
［11］ PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation [C]// Proceedings of the Conference on Empirical Methods in Natural Language Processing. Doha, Qatar: ACL, 2014: 1532-1543.
［12］ HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780.
［13］ RAN Xiangdong, SHAN Zhiguang, FANG Yufei, et al. An LSTM-based method with attention mechanism for travel time prediction [J]. Sensors, 2019, 19(4): 861.
［14］ GRAVES A, SCHMIDHUBER J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures [J]. Neural Networks, 2005, 18(5/6): 602-610.
［15］ BIN Yi, YANG Yang, SHEN Fumin, et al. Describing video with attention-based bidirectional LSTM [J]. IEEE Transactions on Cybernetics, 2019, 49(7): 2631-2641.
［16］ 刘欢，张智雄，王宇飞．BERT模型的主要优化改进方法研究综述［J］．数据分析与知识发现，2021，5（1）：3-15.
LIU Huan,ZHANG Zhixiog,WANG Yufei.A review on main optimization methods of BERT [J]. Data Analysis and Knowledge Discovery, 2021, 5(1): 3-15.(in Chinese)
［17］ 方晓东，刘昌辉，王丽亚，等．基于BERT的复合网络模型的中文文本分类［J］．武汉工程大学学报，2020，42（6）：688-692.
FANG Xiaodong, LIU Changhui, WANG Liya, et al. Chinese text classification based on BERT’s composite network model [J]. Journal of Wuhan Institute of Technology, 2020, 42(6): 688-692.(in Chinese)
［18］ 段丹丹，唐加山，温勇，等．基于BERT模型的中文短文本分类算法［J］．计算机工程，2021，47（1）：79-86.
DUAN Dandan, TANG Jiashan, WEN Yong, et al. Chinese short text classification algorithm based on BERT model [J]. Computer Engineering, 2021, 47(1): 79-86.(in Chinese)
［19］ DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of NAACL-HLT. Minneapolis, USA: ACL, 2019: 4171-4186.
［20］ SU Jing, DAI Qingyun, GUERIN F, et al. BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling [J]. Computer Speech & Language, 2021, 67: 1-14.
［21］ 夏林中，罗德安，刘俊，等．基于注意力机制的双层LSTM自动作文评分系统［J］．深圳大学学报理工版，2020，37（6）：559-566．
XIA Linzhong, LUO De’an, LIU Jun, et al. Attention-based two-layer long short-term memory model for automatic essay scoring [J]. Journal of Shenzhen University Science and Engineering, 2020, 37(6): 559-566.(in Chinese)

相似文献/References:

[1]刘进忙,倪鹏,李超,等.复杂战场环境下的分坐标处理[J].深圳大学学报理工版,2014,31(2):138.[doi:10.3724/SP.J.1249.2014.02138]
　Liu Jinmang,Ni Peng,Li Chao,et al.The independent coordinate processing in complex battlefield[J].Journal of Shenzhen University Science and Engineering,2014,31(3):138.[doi:10.3724/SP.J.1249.2014.02138]
[2]贺成龙,秦洪,于永生.一种空中目标航迹的自适应跟踪算法[J].深圳大学学报理工版,2014,31(4):361.[doi:10.3724/SP.J.1249.2014.04361]
　He Chenglong,Qin Hong,and Yu Yongsheng.An adaptive tracking algorithm for aerial target[J].Journal of Shenzhen University Science and Engineering,2014,31(3):361.[doi:10.3724/SP.J.1249.2014.04361]
[3]陈星宇,黄俊文,周展,等.基于本体论的大数据下用户需求表征[J].深圳大学学报理工版,2017,34(2):173.
　Chen Xingyu,Huang Junwen,Zhou Zhan,et al. Ontology-based user needs representation in the big data context[J].Journal of Shenzhen University Science and Engineering,2017,34(3):173.
[4]陈星宇,周展,黄俊文,等.基于关键词挖掘的客户细分方法[J].深圳大学学报理工版,2017,34(3):300.[doi:10.3724/SP.J.1249.2017.03300]
　Chen Xingyu,Zhou Zhan,Huang Junwen,et al.A keyword-based mining method for customer segmentation[J].Journal of Shenzhen University Science and Engineering,2017,34(3):300.[doi:10.3724/SP.J.1249.2017.03300]
[5]胡布焕,张晶,张凌.一种基于语义相似的中文文档抄袭检测方法[J].深圳大学学报理工版,2020,37(增刊1):107.[doi:10.3724/SP.J.1249.2020.99107]
　HU Buhuan,ZHANG Jing,and ZHANG Ling.A plagiarism detection approach for Chinese documents based on semantic textual similarity[J].Journal of Shenzhen University Science and Engineering,2020,37(3):107.[doi:10.3724/SP.J.1249.2020.99107]
[6]宋昱,等.基于图像块l0梯度最小化的边缘保持平滑算法[J].深圳大学学报理工版,2021,38(3):307.[doi:10.3724/SP.J.1249.2021.03307]
　SONG Yu,and SUN Wenyun,et al.Edge-preserving smoothing algorithm based on l0 gradient minimization of image-patch[J].Journal of Shenzhen University Science and Engineering,2021,38(3):307.[doi:10.3724/SP.J.1249.2021.03307]

备注/Memo

备注/Memo:
Received: 2021-09-19; Accepted: 2021-11-01; Online (CNKI): 2022-04-13
Foundation: Science and Technology Platform Construction for Universities of Department of Education of Guangdong Province (2020KTSCX301); Shenzhen Basic Research Foundation (JCYJ20190808093001772); Special Support Program of Leading Professional (Outstanding Teacher) of National High-Level Personnel of China (ZTZ [2018] No.6)
Corresponding author: Associate professor XIA Linzhong.E-mail: 43966506@qq.com
Citation: XIA Linzhong, YE Jianfeng, LUO De’an, et al. Short text automatic scoring system based on BERT-BiLSTM model [J]. Journal of Shenzhen University Science and Engineering, 2022, 39(3): 349-354.(in Chinese)
基金项目：广东省教育厅高校科研平台资助项目（2020KTSCX301）；深圳市基础研究计划资助项目（JCYJ20190808 093001772）；国家高层次人才特殊支持计划领军人才（教学名师）资助项目（组厅字［2018］6号）
作者简介：夏林中（1980—），深圳信息职业技术学院副教授、博士. 研究方向：自然语言处理. E-mail: 43966506@qq.com
引文：夏林中，叶剑锋，罗德安，等.基于BERT-BiLSTM模型的短文本自动评分系统［J］.深圳大学学报理工版，2022，39（3）：349-354.
更新日期/Last Update: 2022-05-30