research-article

A BERT-based Ensemble Model for Chinese News Topic Prediction

Authors:

Tengteng LiuAuthors Info & Claims

BDE '20: Proceedings of the 2020 2nd International Conference on Big Data Engineering

Pages 18 - 23

https://doi.org/10.1145/3404512.3404524

Published: 05 July 2020 Publication History

Abstract

With the rapid development of big data mining technology in the Chinese commercial field, the news topic prediction becomes increasingly important. Since the accuracy of Chinese news topic classification can directly affect the personalized recommendation effect of the Chinese news system and then affect business profits, the news category prediction performance needs to be higher as possible. With the great success of the BERT model in the past two years, using the BERT model alone has achieved extremely good performance on Chinese text classification tasks. Therefore, using the advantages of the BERT to study more effective methods for the Chinese news classification will become more meaningful. In this paper, we propose a model that combines the advantages of both BERT and the long short-term memory (LSTM) network, named BERT ensemble LSTM-BERT(BERT-LB). Our method is more effective than using BERT alone. This model uses a three-step method to calculate and integrate Chinese news text features. Besides, we use two datasets to evaluate our method and other baseline methods. We demonstrate that the proposed method has the promising ability to predict Chinese news topics and prove its generalization ability.

References

[1]

Lin, F. R., and Liang, C. H. 2008. Storyline-based summarization for news topic retrospection. Decision Support Systems, 45(3), 473--490.

Digital Library

[2]

Newman, D., Chemudugunta, C., Smyth, P., & Steyvers, M. (2006, May). Analyzing entities and topics in news articles using statistical topic models. In International conference on intelligence and security informatics (pp. 93--104). Springer, Berlin, Heidelberg.

Digital Library

[3]

Salton, G., Singhal, A., Buckley, C., & Mitra, M. (1996, March). Automatic text decomposition using text segments and text themes. In Proceedings of the the seventh ACM conference on Hypertext (pp. 53--65).

Digital Library

[4]

Sista, S., Schwartz, R., Leek, T. R., & Makhoul, J. (2002, March). An algorithm for unsupervised topic discovery from broadcast news stories. In Proceedings of the second international conference on Human Language Technology Research (pp. 110--114). Morgan Kaufmann Publishers Inc.

Digital Library

[5]

Dilrukshi, I., De Zoysa, K., & Caldera, A. (2013, April). Twitter news classification using SVM. In 2013 8th International Conference on Computer Science & Education (pp. 287--291). IEEE.

[6]

Carreira, R., Crato, J. M., Gonçalves, D., & Jorge, J. A. (2004, January). Evaluating adaptive user profiles for news classification. In Proceedings of the 9th international conference on Intelligent user interfaces (pp. 206--212).

Digital Library

[7]

Wu, L., Li, Z., Li, M., Ma, W. Y., & Yu, N. (2007, November). Mutually beneficial learning with application to on-line news classification. In Proceedings of the ACM first Ph. D. workshop in CIKM (pp. 85--92).

Digital Library

[8]

Dutta, R., Jana, B., & Majumder, M. (2019, September). Semantic Similarity and Word-Net Based Web News Classification. In International Conference on Innovation in Modern Science and Technology (pp. 728--735). Springer, Cham.

[9]

Kadhim, A. I. (2019). Survey on supervised machine learning techniques for automatic text classification. Artificial Intelligence Review, 52(1), 273--292.

Digital Library

[10]

Du, J., Gui, L., Xu, R., & He, Y. (2017, November). A convolutional attention model for text classification. In National CCF Conference on Natural Language Processing and Chinese Computing (pp. 183--195). Springer, Cham.

[11]

Miao, F., Zhang, P., Jin, L., & Wu, H. (2018, August). Chinese news text classification based on machine learning algorithm. In 2018 10th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 48--51). IEEE.

[12]

Cui, L., & Shi, Y. (2014). A Method based on One-class SVM for News Recommendation. Procedia Computer Science, 31, 281--290.

[13]

Lu, Z., Liu, W., Zhou, Y., Hu, X., & Wang, B. (2017, November). An effective approach for Chinese news headline classification based on multi-representation mixed model with attention and ensemble learning. In National CCF Conference on Natural Language Processing and Chinese Computing (pp. 339--350). Springer, Cham.

[14]

Liu, J., Xia, C., Yan, H., Xie, Z., & Sun, J. (2019). Hierarchical Comprehensive Context Modeling for Chinese Text Classification. IEEE Access, 7, 154546--154559.

[15]

Zhou, C., Sun, C., Liu, Z., & Lau, F. (2015). A C-LSTM neural network for text classification. arXiv preprint arXiv:1511.08630.

[16]

Zhang, L., & Chen, C. (2016, December). Sentiment classification with convolutional neural networks: An experimental study on a large-scale chinese conversation corpus. In 2016 12th International Conference on Computational Intelligence and Security (CIS) (pp. 165--169). IEEE.

[17]

Han, H., Liu, J., & Liu, G. (2018). Attention-based memory network for text sentiment classification. IEEE Access, 6, 68302--68310.

[18]

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

[19]

Zhou, Y., Xu, B., Xu, J., Yang, L., & Li, C. (2016, October). Compositional recurrent neural networks for chinese short text classification. In 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI) (pp. 137--144). IEEE.

[20]

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998--6008).

[21]

Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555.

[22]

Dilrukshi, I., & de Zoysa, K. (2014). A feature selection method for twitter news classification. International Journal of Machine Learning and Computing, 4(4), 365--370.

[23]

Buabin, E. (2012). Boosted hybrid recurrent neural classifier for text document classification on the Reuters news text corpus. International Journal of Machine Learning and Computing, 2(5), 588--592.

[24]

Lu, R., & Yang, Q. (2012). Trend analysis of news topics on twitter. International Journal of Machine Learning and Computing, 2(3), 327--332.

[25]

Chapelle, O., & Wu, M. (2010). Gradient descent optimization of smoothed information retrieval metrics. Information retrieval, 13(3), P.216--235.

[26]

Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv.1909.11942.

[27]

Souza, F., Nogueira, R., & Lotufo, R. (2019). Portuguese Named Entity Recognition using BERT-CRF. arXiv preprint arXiv:1909.10649.

[28]

Rodrigues Makiuchi, M., Warnita, T., Uto, K., & Shinoda, K. (2019, October). Multimodal Fusion of BERT-CNN and Gated CNN Representations for Depression Detection. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop (pp. 55--63).

Digital Library

[29]

Wang, Z., Ng, P., Ma, X., Nallapati, R., & Xiang, B. (2019). Multi-passage bert: A globally normalized bert model for open-domain question answering. arXiv preprint arXiv:1908.08167.

[30]

Greff, Klaus, Srivastava, Rupesh Kumar, Koutník, Jan, Steunebrink, Bas R., & Schmidhuber, Jürgen. (2015). Lstm: a search space odyssey. IEEE Transactions on Neural Networks & Learning Systems, 28(10), 2222--2232.

[31]

Lipton, Z. C., Berkowitz, J., & Elkan, C. (2015). A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019.

[32]

Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg, & Dean, Jeffrey. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26, 3111--3119.

Digital Library

[33]

Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2015). Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2625--2634).

[34]

Wang, C., Zhang, M., Ma, S., & Ru, L. (2008, April). Automatic online news issue construction in web environment. In Proceedings of the 17th international conference on World Wide Web (pp. 457--466).

Digital Library

[35]

Lv, A., & Luo, T. (2018). Authoritarian Practices in the Digital Age| Asymmetrical Power Between Internet Giants and Users in China. International Journal of Communication, 12, 19.

[36]

Bai, S., Kolter, J. Z., & Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01271.

Cited By

Rosnes DStarke ATrattner C(2024)Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity MetricsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659560(201-211)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659560
Häglund EBjörklund J(2024)AI-Driven Contextual Advertising: Toward Relevant Messaging Without Personal DataJournal of Current Issues & Research in Advertising10.1080/10641734.2024.233493945:3(301-319)Online publication date: 29-Apr-2024
https://doi.org/10.1080/10641734.2024.2334939
Starke ASolberg VØverhaug STrattner C(2024)Examining the merits of feature-specific similarity functions in the news domain using human judgmentsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09412-234:4(995-1042)Online publication date: 7-Aug-2024
https://doi.org/10.1007/s11257-024-09412-2
Show More Cited By

Index Terms

A BERT-based Ensemble Model for Chinese News Topic Prediction
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Chinese text classification by the Naïve Bayes Classifier and the associative classifier with multiple confidence threshold values

Each type of classifier has its own advantages as well as certain shortcomings. In this paper, we take the advantages of the associative classifier and the Naive Bayes Classifier to make up the shortcomings of each other, thus improving the accuracy of ...
Chinese News Text Multi Classification Based on Naive Bayes Algorithm
ISCSIC '18: Proceedings of the 2nd International Symposium on Computer Science and Intelligent Control

With the development of Internet, there are more and more text data appear, the companies face the challenge to organize the content and the users feel confused about what is useful content for them. If the text data can be classified will make a ...
Text classification based on data partitioning and parameter varying ensembles
SAC '05: Proceedings of the 2005 ACM symposium on Applied computing

Support vector machines (SVM) are among the best text classifiers so far. Meantimes, ensembles of classifiers are proven to be effective on many domains. It is expected that ensembles of SVM classifiers could achieve better performance. In this paper ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

BDE '20: Proceedings of the 2020 2nd International Conference on Big Data Engineering

May 2020

146 pages

ISBN:9781450377225

DOI:10.1145/3404512

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

BDE 2020

BDE 2020: 2020 2nd International Conference on Big Data Engineering

May 29 - 31, 2020

Shanghai, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
480
Total Downloads

Downloads (Last 12 months)62
Downloads (Last 6 weeks)2

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rosnes DStarke ATrattner C(2024)Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity MetricsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659560(201-211)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659560
Häglund EBjörklund J(2024)AI-Driven Contextual Advertising: Toward Relevant Messaging Without Personal DataJournal of Current Issues & Research in Advertising10.1080/10641734.2024.233493945:3(301-319)Online publication date: 29-Apr-2024
https://doi.org/10.1080/10641734.2024.2334939
Starke ASolberg VØverhaug STrattner C(2024)Examining the merits of feature-specific similarity functions in the news domain using human judgmentsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09412-234:4(995-1042)Online publication date: 7-Aug-2024
https://doi.org/10.1007/s11257-024-09412-2
Abarna SSheeba JPradeep Devaneyan S(2023)A novel ensemble model for identification and classification of cyber harassment on social media platformJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23034645:1(13-36)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-230346
Zou HWang Z(2023)A semi-supervised short text sentiment classification method based on improved Bert model from unlabelled dataJournal of Big Data10.1186/s40537-023-00710-x10:1Online publication date: 15-Mar-2023
https://doi.org/10.1186/s40537-023-00710-x
Syahputra MKemala ATjan FSusanto R(2023)Clickbait Detection in Indonesia Headline News Using BERT Ensemble Models2023 6th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI)10.1109/ISRITI60336.2023.10467417(475-479)Online publication date: 11-Dec-2023
https://doi.org/10.1109/ISRITI60336.2023.10467417
Sagama YAlamsyah A(2023)Multi-Label Classification of Indonesian Online Toxicity using BERT and RoBERTa2023 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT)10.1109/IAICT59002.2023.10205892(143-149)Online publication date: 13-Jul-2023
https://doi.org/10.1109/IAICT59002.2023.10205892
Huang KLi XLiu FYang XYu W(2022)ML-GAT:A Multilevel Graph Attention Model for Stock PredictionIEEE Access10.1109/ACCESS.2022.319900810(86408-86422)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3199008
Miles SYao LMeng WBlack CMiled Z(2022)Comparing PSO-based clustering over contextual vector embeddings to modern topic modelingInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10292159:3Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1016/j.ipm.2022.102921
Sharma RMorwal SAgarwal B(2022)Named entity recognition using neural language model and CRF for Hindi languageComputer Speech and Language10.1016/j.csl.2022.10135674:COnline publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.csl.2022.101356
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents