Abstract
Core Information Extraction (CIE) from web pages aims to extract valuable text to provide data for downstream Text Data Mining (TDM) tasks. Web page representations in existing CIE methods are either based on HTML structural features or visual features. Neither of these representations really understands the semantic associations inherent in the web page, leading to poor extraction quality. This paper proposes a new web page representation method based on semantic features from the perspective of readers’ reading and understanding of web pages. In this method, we introduce a new concept of web page skeleton to parse and represent the web page from a semantic point of view. To observe the relationship between the various parts of the skeleton, we project the skeleton onto the DOM tree and get the skeleton tree. Based on this new web page representation, we propose SCIEnt, a semantic-feature-based web page CIE framework. SCIEnt consists of four modules, i.e. Skeleton Tree Construction, Node Splitting, Node Classification, Semantic Aggregation and Correction. Algorithms in each module can be flexibly replaced according to the requirement of downstream TDM task. We evaluate SCIEnt in terms of three well-studied datasets. Results show that SCIEnt far outperforms baseline methods, and the semantic-feature-based web page representation have superiority in web page CIE.
This work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDC02060200.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Yu, X., Jin, Z.: Web content information extraction based on DOM tree and statistical information. In: 2017 IEEE 17th International Conference on Communication Technology (ICCT), pp. 1308–1311. IEEE (2017)
Yuliana, O.Y., Chang, C.H.: A novel alignment algorithm for effective web data extraction from singleton-item pages. Appl. Intell. 48(11), 4355–4370 (2018). https://doi.org/10.1007/s10489-018-1208-0
Sun, F., Song, D., Liao, L.: DOM based content extraction via text density. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 245–254 (2011)
Tan, Z., He, C., Fang, Y., et al.: Title-based extraction of news contents for text mining. IEEE Access 6, 64085–64095 (2018)
Yao, J., Zuo, X.: A machine learning approach to webpage content exraction (2013)
Utiu, N., Ionescu, V.S.: Learning web content extraction with DOM features. In: 2018 IEEE 14th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 5–11. IEEE (2018)
Yang, H., Lu, H., Li, S., Li, M., Sun, Y.: Research on content extraction of rich text web pages. In: Sun, X., Pan, Z., Bertino, E. (eds.) ICAIS 2019. LNCS, vol. 11635, pp. 279–287. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24268-8_26
Vogels, T., Ganea, O.-E., Eickhoff, C.: Web2Text: deep structured boilerplate removal. In: Pasi, G., Piwowarski, B., Azzopardi, L., Hanbury, A. (eds.) ECIR 2018. LNCS, vol. 10772, pp. 167–179. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-76941-7_13
Leonhardt, J., Anand, A., Khosla, M.: Boilerplate removal using a neural sequence labeling model. In: Companion Proceedings of the Web Conference, pp. 226–229 (2020)
Wang, Q., Fang, Y., Ravula, A., et al.: WebFormer: the web-page transformer for structure information extraction. In: Proceedings of the ACM Web Conference, pp. 3124–3133 (2022)
Cai, D., Yu, S., Wen, J.-R., Ma, W.-Y.: Extracting content structure for web pages based on visual representation. In: Zhou, X., Orlowska, M.E., Zhang, Y. (eds.) APWeb 2003. LNCS, vol. 2642, pp. 406–417. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36901-5_42
Zhao, H., Meng, W., Wu, Z., et al.: Fully automatic wrapper generation for search engines. In: Proceedings of the 14th International Conference on World Wide Web, pp. 66–75 (2005)
Lu, Y., He, H., Zhao, H., et al.: Annotating search results from web databases. IEEE Trans. Knowl. Data Eng. 25(3), 514–527 (2011)
Anderson, N., Hong, J.: Visually extracting data records from the deep web. In: Proceedings of the 22nd International Conference on World Wide Web, pp. 1233–1238 (2013)
Liu, J., Lin, L., Cai, Z., et al.: Deep web data extraction based on visual information processing. J. Ambient Intell. Human. Comput. 1–11 (2017). https://doi.org/10.1007/s12652-017-0587-0
Kumar, A., Morabia, K., Wang, J., et al.: CoVA: context-aware visual attention for webpage information extraction. arXiv preprint arXiv:2110.12320, 2021
Devlin, J., Chang, M.W., Lee, K., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Baroni, M., Chantree, F., Kilgarriff, A., et al.: Cleaneval: a competition for cleaning web pages. In: Lrec (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Z. et al. (2023). SCIEnt: A Semantic-Feature-Based Framework for Core Information Extraction from Web Pages. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13625. Springer, Cham. https://doi.org/10.1007/978-3-031-30111-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-30111-7_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30110-0
Online ISBN: 978-3-031-30111-7
eBook Packages: Computer ScienceComputer Science (R0)