Article

Page-level template detection via isotonic smoothing

Authors:

Deepayan Chakrabarti,

Ravi Kumar, and

Kunal PuneraAuthors Info & Claims

WWW '07: Proceedings of the 16th international conference on World Wide Web

May 2007

Pages 61 - 70

https://doi.org/10.1145/1242572.1242582

Published: 08 May 2007 Publication History

Abstract

We develop a novel framework for the page-level template detection problem. Our framework is built on two main ideas. The first is theautomatic generation of training data for a classifier that, given apage, assigns a templateness score to every DOM node of the page. The second is the global smoothing of these per-node classifier scores bysolving a regularized isotonic regression problem; the latter follows from a simple yet powerful abstraction of templateness on a page. Our extensive experiments on human-labeled test data show that our approachdetects templates effectively.

References

[1]

S. Angelov, B. Harb, S. Kannan, and L.-S. Wang. Weighted isotonic regression under the l₁ norm. In Proc. 17th SODA, pages 783--791, 2006.

Digital Library

[2]

S. Baluja. Browsing on small screens: Recasting web-page segmentation into an efficient machine learning framework. In Proc. 15th WWW, pages 33--42, 2006.

Digital Library

[3]

Z. Bar-Yossef and S. Rajagopalan. Template detection via data mining and its applications. In Proc. 11th WWW, pages 580--591, 2002.

Digital Library

[4]

K. Bharat, A. Broder, J. Dean, and M.R. Henzinger. A comparison of techniques to find mirrored hosts on the WWW. JASIS, 51(12):1114--1122, 2000.

Digital Library

[5]

A.Z. Broder, S.C. Glassman, M.S. Manasse, and G. Zweig. Syntactic clustering of the web. WWW6 / Computer Networks, 29(8-13):1157--1166, 1997.

Digital Library

[6]

S. Chakrabarti, B.E. Dom, and P. Indyk. Enhanced hypertext categorization using hyperlinks. In Proc. SIGMOD, pages 307--318, 1998.

Digital Library

[7]

Y. Chen, X. Xie, W.-Y. Ma, and H.-J. Zhang. Adapting web pages for small-screen devices. Internet Computing, 9(1):50--56, 2005.

Digital Library

[8]

J. Cho, N. Shivakumar, and H. Garcia-Molina. Finding replicated web collections. In Proc. SIGMOD, pages 355--366, 2000.

Digital Library

[9]

B. Davison. Recognizing nepotistic links on the web. In AAAI-2000 Workshop on Artificial Intelligence for Web Search, pages 23--28, 2000.

[10]

S. Debnath, P. Mitra, N. Pal, and C.L. Giles. Automatic identification of informative sections of web pages. TKDE, 17(9):1233--1246, 2005.

Digital Library

[11]

D. Gibson, K. Punera, and A. Tomkins. The volume and evolution of web page templates. In Proc. 14th WWW (Special interest tracks and posters), pages 830--839, 2005.

Digital Library

[12]

L. Hubert and P. Arabie. Comparing partitions. J. Classification, 2:193--218, 1985.

[13]

H.-Y. Kao, M.-S. Chen, S.-H. Lin, and J.-M. Ho. Entropy-based link analysis for mining web informative structures. In Proc. 11th CIKM, pages 574--581, 2002.

Digital Library

[14]

H.-Y. Kao, J.-M. Ho, and M.-S. Chen. WISDOM: Web intrapage informative structure mining based on document object model. TKDE, 17(5):614--627, 2005.

Digital Library

[15]

R. Kumar, K. Punera, and A. Tomkins. Hierarchical topic segmentation of websites. In Proc. 12th KDD, pages 257--266, 2006.

Digital Library

[16]

N. Kushmerick. Learning to remove internet advertisement. In Proc. 3rd Agents, pages 175--181, 1999.

Digital Library

[17]

G. Milligan and M. Cooper. A study of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research, 21(4):441--458, 1986.

[18]

T. Mitchell. Machine Learning. McGraw Hill, 1997.

Digital Library

[19]

T. Morton-Jones, P. Diggle, L. Parker, H.O. Dickinson, and K. Blinks. Additive isotonic regression models in epidemiology. Statistics in Medicine, 19(6):849--859, 2000.

[20]

P.M. Pardalos and G. Xue. Algorithms for a class of isotonic regression problems. Algorithmica, 23(3):211--222, 1999.

[21]

T. Robertson, F.T. Wright, and R.L. Dykstra. Order-Restrictied Statistical Inference. Wiley, 1988.

[22]

R. Song, H. Liu, J.-R. Wen, and W.-Y. Ma. Learning block importance models for web pages. In Proc. 13th WWW, pages 203--211, 2004.

Digital Library

[23]

Q. Stout. Optimal algorithms for unimodal regression. Computing Science and Statistics, 32:348--355, 2000.

[24]

K. Vieira, A. Silva, N. Pinto, E. Moura, J. Cavalcanti, and J. Freire. A fast and robust method for web page template detection and removal. In Proc. 15th CIKM, pages 256--267, 2006.

Digital Library

[25]

L. Yi and B. Liu. Web page cleaning for web mining through feature weighting. In Proc. 18th IJCAI, pages 43--50, 2003.

Digital Library

[26]

L. Yi, B. Liu, and X. Li. Eliminating noisy information in web pages for data mining. In Proc. 9th KDD, pages 296--305, 2003.

Digital Library

[27]

X. Yin and W.S. Lee. Using link analysis to improve layout on mobile devices. In Proc. 13th WWW, pages 338--344, 2004.

Digital Library

Cited By

Leonhardt JAnand AKhosla M(2020)Boilerplate Removal using a Neural Sequence Labeling ModelCompanion Proceedings of the Web Conference 202010.1145/3366424.3383547(226-229)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366424.3383547
Uzun E(2020)A Novel Web Scraping Approach Using the Additional Information Obtained From Web PagesIEEE Access10.1109/ACCESS.2020.29845038(61726-61740)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2984503
Jiang ZYin HWu YLyu YMin GZhang X(2019)Constructing Novel Block Layouts for Webpage AnalysisACM Transactions on Internet Technology10.1145/332645719:3(1-18)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.1145/3326457
Show More Cited By

Index Terms

Page-level template detection via isotonic smoothing
1. Information systems
  1. Information systems applications

Recommendations

Isotonic Median Regression: A Linear Programming Approach

The isotonic median regression problem arises in statistics. It is known that the isotonic median regression problem, with respect to a complete order, may be solved by a “Pool Adjacent Violators” algorithm. In this paper we show that this algorithm is ...
Read More
Effectiveness of template detection on noise reduction and websites summarization

The World Wide Web is the most rapidly growing and accessible source of information. Its popularity has been largely influenced by the wide availability of the Internet in almost every modern house and even on the go after the wide-spread of the ...
Read More
Template detection for large scale search engines
SAC '06: Proceedings of the 2006 ACM symposium on Applied computing

Templates in web sites hurt search engine retrieval performance, especially in content relevance and link analysis. Current template removal methods suffer from processing speed and scalability when dealing with large volume web pages. In this paper, we ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '07: Proceedings of the 16th international conference on World Wide Web

May 2007

1382 pages

ISBN:9781595936547

DOI:10.1145/1242572

General Chairs:
Carey Williamson
University of Calgary, Canada
,
Mary Ellen Zurko
IBM, USA
,
Program Chairs:
Peter Patel-Schneider
Bell Labs Research, USA
,
Prashant Shenoy
University of Massachusetts at Amherst, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW'07

Sponsor:

ACM

WWW'07: 16th International World Wide Web Conference

May 8 - 12, 2007

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
708
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Leonhardt JAnand AKhosla M(2020)Boilerplate Removal using a Neural Sequence Labeling ModelCompanion Proceedings of the Web Conference 202010.1145/3366424.3383547(226-229)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366424.3383547
Uzun E(2020)A Novel Web Scraping Approach Using the Additional Information Obtained From Web PagesIEEE Access10.1109/ACCESS.2020.29845038(61726-61740)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2984503
Jiang ZYin HWu YLyu YMin GZhang X(2019)Constructing Novel Block Layouts for Webpage AnalysisACM Transactions on Internet Technology10.1145/332645719:3(1-18)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.1145/3326457
Alarte JSilva JTamarit S(2019)What Web Template Extractor Should I Use? A Benchmarking and Comparison for Five Template ExtractorsACM Transactions on the Web10.1145/331681013:2(1-19)Online publication date: 27-Mar-2019
https://dl.acm.org/doi/10.1145/3316810
Chen YYao Z(2019)Multi-layer Filtering Webpage Classification Method Based on SVMHuman Centered Computing10.1007/978-3-030-37429-7_56(554-559)Online publication date: 12-Dec-2019
https://doi.org/10.1007/978-3-030-37429-7_56
Vogels TGanea OEickhoff C(2018)Web2Text: Deep Structured Boilerplate RemovalAdvances in Information Retrieval10.1007/978-3-319-76941-7_13(167-179)Online publication date: 1-Mar-2018
https://doi.org/10.1007/978-3-319-76941-7_13
Uçar EUzun ETüfekci P(2017)A novel algorithm for extracting the user reviews from web pagesJournal of Information Science10.1177/016555151666644643:5(696-712)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1177/0165551516666446
Omari AKimelfeld BYahav EShoham SKrishnapuram BShah MSmola AAggarwal CShen DRastogi R(2016)Lossless Separation of Web Pages into Layout Code and DataProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/2939672.2939858(1805-1814)Online publication date: 13-Aug-2016
https://dl.acm.org/doi/10.1145/2939672.2939858
Freire de Amorim E(2015)HTML Segmentation for Different Types of Web PagesThe Evolution of the Internet in the Business Sector10.4018/978-1-4666-7262-8.ch005(98-119)Online publication date: 2015
https://doi.org/10.4018/978-1-4666-7262-8.ch005
Madaan AChu W(2015)In-depth querying of web-based medical documentsInternational Journal of Computational Science and Engineering10.1504/IJCSE.2015.07265011:3(284-296)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1504/IJCSE.2015.072650
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents