Article

A statistical approach for efficient crawling of rich internet applications

Authors:

Mustafa Emre Dincturk,

Suryakant Choudhary,

Gregor von Bochmann,

Guy-Vincent Jourdan,

Iosif Viorel OnutAuthors Info & Claims

ICWE'12: Proceedings of the 12th international conference on Web Engineering

Pages 362 - 369

https://doi.org/10.1007/978-3-642-31753-8_29

Published: 23 July 2012 Publication History

Publisher Site

Abstract

Modern web technologies, like AJAX result in more responsive and usable web applications, sometimes called Rich Internet Applications (RIAs). Traditional crawling techniques are not sufficient for crawling RIAs. We present a new strategy for crawling RIAs. This new strategy is designed based on the concept of "Model-Based Crawling" introduced in [3] and uses statistics accumulated during the crawl to select what to explore next with a high probability of uncovering some new information. The performance of our strategy is compared with our previous strategy, as well as the classical Breadth-First and Depth-First on two real RIAs and two test RIAs. The results show this new strategy is significantly better than the Breadth-First and the Depth-First strategies (which are widely used to crawl RIAs), and outperforms our previous strategy while being much simpler to implement.

References

[1]

Bau, J., Bursztein, E., Gupta, D., Mitchell, J.C.: State of the Art: Automated Black-Box Web Application Vulnerability Testing. In: Proc. IEEE Symposium on Security and Privacy (2010)

Digital Library

Google Scholar

[2]

Benjamin, K., von Bochmann, G., Jourdan, G. V., Onut, I. V.: Some Modeling Challenges when Testing Rich Internet Applications for Security. In: First International Workshop on Modeling and Detection of Vulnerabilities, Paris, France (2010)

Digital Library

Google Scholar

[3]

Benjamin, K., von Bochmann, G., Dincturk, M. E., Jourdan, G.-V., Onut, I. V.: A Strategy for Efficient Crawling of Rich Internet Applications. In: Auer, S., Díaz, O., Papadopoulos, G. A. (eds.) ICWE 2011. LNCS, vol. 6757, pp. 74-89. Springer, Heidelberg (2011)

Digital Library

Google Scholar

[4]

Carpento, G., Dell'amico, M., Toth, P.: Exact solution of large-scale, asymmetric traveling salesman problems. ACM Trans. Math. Softw. 21(4) (1995)

Digital Library

Google Scholar

[5]

W3C. Document Object Model, DOM (2005), http://www.w3.org/DOM/

Google Scholar

[6]

Garrett, J. J.: Adaptive Path (2005), http://www.adaptivepath.com/publications/essays/archives/000385.php

Google Scholar

Cited By

View all

Fayzrakhmanov RSallinger ESpencer BFurche TGottlob GChampin PGandon FMédini LLalmas MIpeirotis P(2018)Browserless Web Data ExtractionProceedings of the 2018 World Wide Web Conference10.1145/3178876.3186008(1095-1104)Online publication date: 10-Apr-2018
https://dl.acm.org/doi/10.1145/3178876.3186008
Beroual OGuérin FHallé SGaleotti JPetke J(2017)Searching for behavioural bugs with stateful test oracles in web crawlersProceedings of the 10th International Workshop on Search-Based Software Testing10.5555/3105427.3105430(7-13)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.5555/3105427.3105430
Choudhary SDincturk EMirtaheri SBochmann GJourdan GOnut I(2014)Model-based rich internet applications crawlingJournal of Web Engineering10.5555/2685119.268512413:3-4(243-262)Online publication date: 1-Jul-2014
https://dl.acm.org/doi/10.5555/2685119.2685124
Show More Cited By

Recommendations

A Model-Based Approach for Crawling Rich Internet Applications

New Web technologies, like AJAX, result in more responsive and interactive Web applications, sometimes called Rich Internet Applications (RIAs). Crawling techniques developed for traditional Web applications are not sufficient for crawling RIAs. The ...
A strategy for efficient crawling of rich internet applications
ICWE'11: Proceedings of the 11th international conference on Web engineering

New web application development technologies such as Ajax, Flex or Silverlight result in so-called Rich Internet Applications (RIAs) that provide enhanced responsiveness, but introduce new challenges for crawling that cannot be addressed by the ...
Dist-RIA Crawler: A Distributed Crawler for Rich Internet Applications
3PGCIC '13: Proceedings of the 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing

Crawling web applications is important for indexing, accessibility and security assessment. Crawling traditional web applications is an old problem, as old as the web itself. Crawling Rich Internet Applications (RIA) quickly and efficiently, however, is ...

Comments

Information & Contributors

Information

Published In

ICWE'12: Proceedings of the 12th international conference on Web Engineering

July 2012

511 pages

ISBN:9783642317521

Editors:
Marco Brambilla
Dipartimento di Elettronica e Informazione, Politecnico di Milano, Via Ponzio 34/5, Milano, Italy
,
Takehiro Tokuda
Department of Computer Science, Tokyo Institute of Technology, 2-12-1 Oookayama, Tokyo, Japan
,
Robert Tolksdorf
Institut für Informatik, Freie Universität Berlin, Königin-Luise-Strasse 24-26, Berlin, Germany

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 July 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fayzrakhmanov RSallinger ESpencer BFurche TGottlob GChampin PGandon FMédini LLalmas MIpeirotis P(2018)Browserless Web Data ExtractionProceedings of the 2018 World Wide Web Conference10.1145/3178876.3186008(1095-1104)Online publication date: 10-Apr-2018
https://dl.acm.org/doi/10.1145/3178876.3186008
Beroual OGuérin FHallé SGaleotti JPetke J(2017)Searching for behavioural bugs with stateful test oracles in web crawlersProceedings of the 10th International Workshop on Search-Based Software Testing10.5555/3105427.3105430(7-13)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.5555/3105427.3105430
Choudhary SDincturk EMirtaheri SBochmann GJourdan GOnut I(2014)Model-based rich internet applications crawlingJournal of Web Engineering10.5555/2685119.268512413:3-4(243-262)Online publication date: 1-Jul-2014
https://dl.acm.org/doi/10.5555/2685119.2685124
Dincturk MJourdan GBochmann GOnut I(2014)A Model-Based Approach for Crawling Rich Internet ApplicationsACM Transactions on the Web10.1145/26263718:3(1-39)Online publication date: 8-Jul-2014
https://dl.acm.org/doi/10.1145/2626371
Mirtaheri SDinçtürk MHooshmand SBochmann GJourdan GOnut INg JCordy JCzarnecki K(2013)A brief history of web crawlersProceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research10.5555/2555523.2555529(40-54)Online publication date: 18-Nov-2013
https://dl.acm.org/doi/10.5555/2555523.2555529
Kordomatis IHerzog CFayzrakhmanov RKrüpl-Sypien BHolzinger WBaumgartner RCamacho DAkerkar RRodriguez Moreno M(2013)Web object identification for web automation and meta-searchProceedings of the 3rd International Conference on Web Intelligence, Mining and Semantics10.1145/2479787.2479798(1-12)Online publication date: 12-Jun-2013
https://dl.acm.org/doi/10.1145/2479787.2479798
Choudhary SDincturk MMirtaheri SJourdan GBochmann GOnut I(2013)Building rich internet applications modelsProceedings of the 13th international conference on Web Engineering10.1007/978-3-642-39200-9_25(291-305)Online publication date: 8-Jul-2013
https://dl.acm.org/doi/10.1007/978-3-642-39200-9_25
Choudhary SDincturk MMirtaheri SMoosavi Avon Bochmann GJourdan GOnut INg JJacobsen HZou Y(2012)Crawling rich internet applicationsProceedings of the 2012 Conference of the Center for Advanced Studies on Collaborative Research10.5555/2399776.2399790(146-160)Online publication date: 5-Nov-2012
https://dl.acm.org/doi/10.5555/2399776.2399790

Abstract

References

Cited By

Recommendations

A Model-Based Approach for Crawling Rich Internet Applications

A strategy for efficient crawling of rich internet applications

Dist-RIA Crawler: A Distributed Crawler for Rich Internet Applications

Comments

Information

Published In

Sponsors

Publisher

Publication History

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations