[1]夏林中,罗德安,刘俊,等.基于注意力机制的双层LSTM自动作文评分系统[J].深圳大学学报理工版,2020,37(6):559-566.[doi:10.3724/SP.J.1249.2020.06559]
 XIA Linzhong,LUO Dean,LIU Jun,et al.Attention-based two-layer long short-term memory model for automatic essay scoring[J].Journal of Shenzhen University Science and Engineering,2020,37(6):559-566.[doi:10.3724/SP.J.1249.2020.06559]
点击复制

基于注意力机制的双层LSTM自动作文评分系统()
分享到:

《深圳大学学报理工版》[ISSN:1000-2618/CN:44-1401/N]

卷:
第37卷
期数:
2020年第6期
页码:
559-566
栏目:
电子与信息科学
出版日期:
2020-11-09

文章信息/Info

Title:
Attention-based two-layer long short-term memory model for automatic essay scoring
文章编号:
202006002
作者:
夏林中罗德安刘俊管明祥张振久龚爱平
深圳信息职业技术学院人工智能技术应用工程实验室,广东深圳 518172
Author(s):
XIA Linzhong LUO De’an LIU Jun GUAN Mingxiang ZHANG Zhenjiu and GONG Aiping
Engineering Applications of Artificial Intelligence Technology Laboratory, Shenzhen Institute of Information Technology, Shenzhen 518172, Guangdong Province, P.R.China
关键词:
人工智能 自然语言处理 自动作文评分 长短时记忆网络 注意力机制 二次加权kappa系数
Keywords:
artificial intelligence natural language processing automatic essay scoring long short-term memory attention mechanism quadratic weighted kappa coefficient
分类号:
TP18
DOI:
10.3724/SP.J.1249.2020.06559
文献标志码:
A
摘要:
研究一种基于新型神经网络结构的自动作文评分模型,该模型包括双层长短时记忆(two-layer long short-term memory, LSTM)神经网络层和注意力机制层,模型输入层的词向量通过word embedding预训练谷歌文本库生成.相较于基于本地文本数据集预训练,预训练谷歌文本库生成的词向量含有更丰富的上下文语义信息及依赖关系;双层长短时记忆网络的下层抽取上下文语义信息及隐藏的上下文依赖关系,上层捕获更深层次的上下文依赖关系;注意力机制依据双层长短时记忆网络的输出计算注意力概率,以突出关键信息在文本中的重要程度.模型所使用数据集由Hewlett基金提供,并以二次加权kappa系数作为模型的评估指标.实验结果表明,对比其他基准模型(如双向LSTM模型和SKIPFLOW-LSTM模型等),基于注意力机制的双层LSTM模型所获二次加权kappa系数平均值最好.
Abstract:
We propose a neural network architecture-based automatic essay scoring model which contains a two-layer long short-term memory (LSTM) and an attention mechanism layer. The Google word vector dataset, which includes the richer word information and contextual information than the local-trained word vector dataset, is used to generate the embedding word vector of the input layer of the model by pre-training. The lower layer of the two-layer LSTM network captures the context semantic information and hidden context dependency, and the upper layer extracts the deeper context dependency. The attention mechanism layer focuses on the information extracted from the upper hidden layer of two-layer LSTM and calculates the attention probability to highlight the importance of key information in the text. The dataset used for automatic essay scoring task is provided by the Hewlett Foundation, and the quadratic weighted kappa coefficient is used as the evaluation index of the model. The experimental results show that the proposed method outperforms other automatic essay scoring baseline models such as bidirectional LSTM, SKIPFLOW-LSTM, and so on, in terms of the value of quadratic weighted kappa coefficient.

参考文献/References:

[1] DIEGO P, ATKINSON J. Coherence-based automatic essay assessment[J]. IEEE Intelligent Systems, 2018, 33(5): 26-36.
[2] DIKLI S. An overview of automated scoring of essays[J]. The Journal of Technology, Learning, and Assessment, 2006, 5(1): 1-35.
[3] ATTALI Y, BURSTEIN J. Automated essay scoring with e-rater@v.2.0[J]. ETS Research Report, 2005, 2004(2): 1-21.
[4] LARKEY L S. Automatic essay grading using text categorization techniques[C]// Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Melbourne, Australia: ACM, 1998: 90-95.
[5] WANG Huilan, ZHANG Keliang, LI Yali, et al. A new linguistic feature for automated essay scoring[C]// Proceedings of the 4th International Universal Communication Symposium. Beijing: IEEE, 2010: 340-344.
[6] CHEN Hongbo, HE Ben, LUO Tiejian, et al. A ranked-based learning approach to automated essay scoring[C]// Proceedings of the Second International Conference on Cloud and Green Computing. Xiangtan, China: IEEE, 2012: 448-455.
[7] YANNAKOUDAKIS H, BRISCOE T, MEDLOCK B. A new dataset and method for automatically grading ESOL texts[C]// Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. Portland, USA: ACL, 2011: 180-189.
[8] ZUPANC K, BOSNI Z. Automated essay evaluation augmented with semantic coherence measures[C]// Proceedings of the International Conference on Data Mining. Shenzhen, China: IEEE, 2014: 1133-1138.
[9] CLAUDIA L, MARTIN C. C-rater: automated scoring of short-answer questions[J]. Computers and the Humanities, 2003, 37(4): 389-405.
[10] BLEI D M, NG A Y, JORDAN M I. Latent Dirichlet allocation[J]. Journal of Machine Learning Research, 2003, 3(4/5): 993-1022.
[11] COLLOBERT R, WESTON J, BOTTOU L, et al. Natural language processing (almost) from scratch[J]. The Journal of Machine and Learning Research, 2011, 12: 2493-2537.
[12] ALI M A, TAN G Z, HUSSAIN A. Bidirectional recurrent neural network approach for Arabic named entity recognition[J]. Future Internet, 2018, 10(12): 123.
[13] LIAO Shiyang, WANG Junbo, YU Ruiyun, et al. CNN for situations understanding based on sentiment analysis of twitter data[J]. Procedia Computer Science, 2017, 111: 376-381.
[14] RAN Xiangdong, SHAN Zhiguang, FANG Yufei, et al. An LSTM-based method with attention mechanism for travel time prediction[J]. Sensors, 2019, 19(4): 861.
[15] NOWAK J, TASPINAR A, SCHERER R. LSTM recurrent neural networks for short text and sentiment classification[C]// Proceedings of the 16th International Conference on Artificial Intelligence and Soft Computing. Zakopane, Poland: Springer, 2017: 553-562.
[16] DUAN Yanjie, LV Yisheng, WANG Feiyue. Travel time prediction with LSTM neural network[C]// Proceedings of the 19th International Conference on Intelligent Transportation Systems. Rio de Janeiro, Brazil: IEEE, 2016: 1053-1058.
[17] GRAVES A, SCHMIDHUBER J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures[J]. Neural Networks, 2005, 18(5/6): 602-610.
[18] BIN Yi, YANG Yang, SHEN Fumin, et al. Describing video with attention-based bidirectional LSTM[J]. IEEE Transactions on Cybernetics, 2019, 49(7): 2631-2641.
[19] YIN Wenpeng, EBERT S, SCHTZE H. Attention-based convolutional neural network for machine comprehension[C]// Proceedings of the Workshop on Human-Computer Question Answering. San Diego, USA, 2016: 15-21.
[20] DONG Fei, ZHANG Yue. Automatic features for essay scoring - an empirical study[C]// Proceedings of the Conference on Empirical Methods in Natural Language Processing. Austin, USA: ACL, 2016: 1072-1077.
[21] TAGHIPOUR K, NG H T. A neural approach to automated essay scoring[C]// Proceedings of the Conference on Empirical Methods in Natural Language Processing. Austin, USA: ACL, 2016: 1882-1891.
[22] DONG Fei, ZHANG Yue, YANG Jie. Attention-based recurrent convolutional neural network for automatic essay scoring[C]// Proceedings of the 21st Conference on Computational Natural Language Learning. Vancouver, Canada: ACL, 2017: 153-162.
[23] TAY Y, PHAN M C, TUAN L A, et al. SKIPFLOW: incorporating neural coherence features for end-to-end automatic text scoring[EB/OL]. arXiv, 2017[2019-12-01]. https://arxiv.org/abs/1711.04981.
[24] CHEN Minping, LI Xia. Relevance-based automated essay scoring via hierarchical recurrent model[C]// Proceedings of the International Conference on Asian Language Processing. Bandung, Indonesia: IEEE, 2018: 378-383.
[25] LIANG Guoxi, ON B, JEONG D, et al. Automated essay scoring: a Siamese bidirectional LSTM neural network architecture[J]. Symmetry, 2018, 10(12): 682.
[26] LEE H, GROSSE R, RANGANATH R, et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations[C]// Proceedings of the 26th Annual International Conference on Machine Learning. Montreal, Canada: ACM, 2009: 609-616.

相似文献/References:

[1]潘长城,徐晨,李国.解全局优化问题的差分进化策略[J].深圳大学学报理工版,2008,25(2):211.
 PAN Chang-cheng,XU Chen,and LI Guo.Differential evolutionary strategies for global optimization[J].Journal of Shenzhen University Science and Engineering,2008,25(6):211.
[2]骆剑平,李霞.求解TSP的改进混合蛙跳算法[J].深圳大学学报理工版,2010,27(2):173.
 LUO Jian-ping and LI Xia.Improved shuffled frog leaping algorithm for solving TSP[J].Journal of Shenzhen University Science and Engineering,2010,27(6):173.
[3]蔡良伟,李霞.基于混合蛙跳算法的作业车间调度优化[J].深圳大学学报理工版,2010,27(4):391.
 CAI Liang-wei and LI Xia.Optimization of job shop scheduling based on shuffled frog leaping algorithm[J].Journal of Shenzhen University Science and Engineering,2010,27(6):391.
[4]张重毅,刘彦斌,于繁华,等.CDA市场环境模型进化研究[J].深圳大学学报理工版,2010,27(4):413.
 ZHANG Zhong-yi,LIU Yan-bin,YU Fan-hua,et al.Research on the evolution model of CDA market environment[J].Journal of Shenzhen University Science and Engineering,2010,27(6):413.
[5]姜建国,周佳薇,郑迎春,等.一种双菌群细菌觅食优化算法[J].深圳大学学报理工版,2014,31(1):43.[doi:10.3724/SP.J.1249.2014.01043]
 Jiang Jianguo,Zhou Jiawei,Zheng Yingchun,et al.A double flora bacteria foraging optimization algorithm[J].Journal of Shenzhen University Science and Engineering,2014,31(6):43.[doi:10.3724/SP.J.1249.2014.01043]
[6]蔡良伟,刘思麒,李霞,等.基于蚁群优化的正则表达式分组算法[J].深圳大学学报理工版,2014,31(3):279.[doi:10.3724/SP.J.1249.2014.03279]
 Cai Liangwei,Liu Siqi,Li Xia,et al.Regular expression grouping algorithm based on ant colony optimization[J].Journal of Shenzhen University Science and Engineering,2014,31(6):279.[doi:10.3724/SP.J.1249.2014.03279]
[7]宁剑平,王冰,李洪儒,等.递减步长果蝇优化算法及应用[J].深圳大学学报理工版,2014,31(4):367.[doi:10.3724/SP.J.1249.2014.04367]
 Ning Jianping,Wang Bing,Li Hongru,et al.Research on and application of diminishing step fruit fly optimization algorithm[J].Journal of Shenzhen University Science and Engineering,2014,31(6):367.[doi:10.3724/SP.J.1249.2014.04367]
[8]刘万峰,李霞.车辆路径问题的快速多邻域迭代局部搜索算法[J].深圳大学学报理工版,2015,32(2):196.[doi:10.3724/SP.J.1249.2015.02000]
 Liu Wanfeng,and Li Xia,A fast multi-neighborhood iterated local search algorithm for vehicle routing problem[J].Journal of Shenzhen University Science and Engineering,2015,32(6):196.[doi:10.3724/SP.J.1249.2015.02000]
[9]蔡良伟,程璐,李军,等.基于遗传算法的正则表达式规则分组优化[J].深圳大学学报理工版,2015,32(3):281.[doi:10.3724/SP.J.1249.2015.03281]
 Cai Liangwei,Cheng Lu,Li Jun,et al.Regular expression grouping optimization based on genetic algorithm[J].Journal of Shenzhen University Science and Engineering,2015,32(6):281.[doi:10.3724/SP.J.1249.2015.03281]
[10]王守觉,鲁华祥,陈向东,等.人工神经网络硬件化途径与神经计算机研究[J].深圳大学学报理工版,1997,14(1):8.
 Wang Shoujue,Lu Huaxiang,Chen Xiangdong and Zeng Yujuan.On the Hardware for Artificial Neural Networks and Neurocomputer[J].Journal of Shenzhen University Science and Engineering,1997,14(6):8.

备注/Memo

备注/Memo:
Received:2019-12-26;Accepted:2020-02-12
Foundation:Shenzhen Science and Technology Basic Research Foundation (JCYJ20190808093001772); Engineering Applications of Artificial Intelligence Technology Laboratory of Shenzhen Institute of Information Technology (PT201701); Research Fund of Shenzhen Institute of Information Technology (ZY201708)
Corresponding author:Associate professor XIA Linzhong.E-mail: 43966506@qq.com
Citation:XIA Linzhong,LUO De’an,LIU Jun,et al.Attention-based two-layer long short-term memory model for automatic essay scoring[J]. Journal of Shenzhen University Science and Engineering, 2020, 37(6): 559-566.(in Chinese)
基金项目:深圳市科技计划基础研究资助项目(JCYJ201908080930 01772);深圳信息职业技术学院人工智能技术应用工程实验室基金资助项目(PT201701);深圳信息职业技术学院校级科研培育资助项目(ZY201708)
作者简介:夏林中(1980—),深圳信息职业技术学院副教授、博士. 研究方向:自然语言处理. E-mail: 43966506@qq.com
引文:夏林中,罗德安,刘俊,等.基于注意力机制的双层LSTM自动作文评分系统[J]. 深圳大学学报理工版,2020,37(6):559-566.
更新日期/Last Update: 2020-11-26