Article

Securing web service by automatic robot detection

Authors:

KyoungSoo Park,

Vivek S. Pai,

Kang-Won Lee,

Seraphin CaloAuthors Info & Claims

ATEC '06: Proceedings of the annual conference on USENIX '06 Annual Technical Conference

Page 23

Published: 30 May 2006 Publication History

Publisher Site

Abstract

Web sites are routinely visited by automated agents known as Web robots, that perform acts ranging from the beneficial, such as indexing for search engines, to the malicious, such as searching for vulnerabilities, attempting to crack passwords, or spamming bulletin boards. Previous work to identify malicious robots has relied on ad-hoc signature matching and has been performed on a per-site basis. As Web robots evolve and diversify, these techniques have not been scaling.

We approach the problem as a special form of the Turing test and defend the system by inferring if the traffic source is human or robot. By extracting the implicit patterns of human Web browsing, we develop simple yet effective algorithms to detect human users. Our experiments with the CoDeeN content distribution network show that 95% of human users are detected within the first 57 requests, and 80% can be identified in only 20 requests, with a maximum false positive rate of 2.4%. In the time that this system has been deployed on CoDeeN, robot-related abuse complaints have dropped by a factor of 10.

Cited By

View all

Arzani BCiraci SSaroiu SWolman AStokes JOuthred GDiwu LBhagwan RPorter G(2020)PrivateEyeProceedings of the 17th Usenix Conference on Networked Systems Design and Implementation10.5555/3388242.3388300(797-816)Online publication date: 25-Feb-2020
https://dl.acm.org/doi/10.5555/3388242.3388300
Lie DManiatis P(2017)GlimmersProceedings of the 16th Workshop on Hot Topics in Operating Systems10.1145/3102980.3102996(94-99)Online publication date: 7-May-2017
https://dl.acm.org/doi/10.1145/3102980.3102996
Pham KSantos AFreire JPerego RSebastiani FAslam JRuthven IZobel J(2016)Understanding Website Behavior based on User AgentProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2914757(1053-1056)Online publication date: 7-Jul-2016
https://dl.acm.org/doi/10.1145/2911451.2914757
Show More Cited By

Index Terms

Securing web service by automatic robot detection

Recommendations

Web Spambot Detection Based on Web Navigation Behaviour
AINA '10: Proceedings of the 2010 24th IEEE International Conference on Advanced Information Networking and Applications

Web robots have been widely used for various beneficial and malicious activities. Web spambots are a type of web robot that spreads spam content throughout the web by typically targeting Web 2.0 applications. They are intelligently designed to replicate ...
A soft computing approach for benign and malicious web robot detection

We propose a method called SMART (Soft computing for MAlicious RoboT detection).The method detects benign and malicious robots, and human visitors to a web server.SMART selects its features on a particular web server by fuzzy rough set theory.A graph-...
Web Bot Detection Evasion Using Deep Reinforcement Learning
ARES '22: Proceedings of the 17th International Conference on Availability, Reliability and Security

Web bots are vital for the web as they can be used to automate several actions, some of which would have otherwise been impossible or very time consuming. These actions can be benign, such as website testing and web indexing, or malicious, such as ...

Comments

Information & Contributors

Information

Published In

ATEC '06: Proceedings of the annual conference on USENIX '06 Annual Technical Conference

May 2006

60 pages

Publisher

USENIX Association

United States

Publication History

Published: 30 May 2006

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
11
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Arzani BCiraci SSaroiu SWolman AStokes JOuthred GDiwu LBhagwan RPorter G(2020)PrivateEyeProceedings of the 17th Usenix Conference on Networked Systems Design and Implementation10.5555/3388242.3388300(797-816)Online publication date: 25-Feb-2020
https://dl.acm.org/doi/10.5555/3388242.3388300
Lie DManiatis P(2017)GlimmersProceedings of the 16th Workshop on Hot Topics in Operating Systems10.1145/3102980.3102996(94-99)Online publication date: 7-May-2017
https://dl.acm.org/doi/10.1145/3102980.3102996
Pham KSantos AFreire JPerego RSebastiani FAslam JRuthven IZobel J(2016)Understanding Website Behavior based on User AgentProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2914757(1053-1056)Online publication date: 7-Jul-2016
https://dl.acm.org/doi/10.1145/2911451.2914757
Doran DGokhale S(2016)An integrated method for real time and offline web robot detectionExpert Systems: The Journal of Knowledge Engineering10.1111/exsy.1218433:6(592-606)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1111/exsy.12184
Chu ZGianvecchio SKoehl AWang HJajodia S(2013)Blog or blockComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2012.10.00557:3(634-646)Online publication date: 1-Feb-2013
https://dl.acm.org/doi/10.1016/j.comnet.2012.10.005
Stevanovic DVlajic NAn A(2013)Detection of malicious and non-malicious website visitors using unsupervised neural network learningApplied Soft Computing10.1016/j.asoc.2012.08.02813:1(698-708)Online publication date: 1-Jan-2013
https://dl.acm.org/doi/10.1016/j.asoc.2012.08.028
Sengupta AMazumdar CBagchi APotdar VMukhopadhyay D(2012)Specification and validation of enterprise information security policiesProceedings of the CUBE International Information Technology Conference10.1145/2381716.2381868(801-808)Online publication date: 3-Sep-2012
https://dl.acm.org/doi/10.1145/2381716.2381868
Hayati PFiroozeh NPotdar VChai KPotdar VMukhopadhyay D(2012)How much money do spammers make from your website?Proceedings of the CUBE International Information Technology Conference10.1145/2381716.2381856(732-739)Online publication date: 3-Sep-2012
https://dl.acm.org/doi/10.1145/2381716.2381856
Stevanovic DAn AVlajic N(2012)Feature evaluation for web crawler detection with data mining techniquesExpert Systems with Applications: An International Journal10.1016/j.eswa.2012.01.21039:10(8707-8717)Online publication date: 1-Aug-2012
https://dl.acm.org/doi/10.1016/j.eswa.2012.01.210
Guha SCheng BFrancis PAndersen DRatnasamy S(2011)PrivadProceedings of the 8th USENIX conference on Networked systems design and implementation10.5555/1972457.1972475(169-182)Online publication date: 30-Mar-2011
https://dl.acm.org/doi/10.5555/1972457.1972475
Show More Cited By

Abstract

Cited By

Index Terms

Recommendations

Web Spambot Detection Based on Web Navigation Behaviour

A soft computing approach for benign and malicious web robot detection

Web Bot Detection Evasion Using Deep Reinforcement Learning

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations