research-article

Empirical comparison of IP reputation databases

Authors:

Jernej Porenta,

Mojca CiglaričAuthors Info & Claims

CEAS '11: Proceedings of the 8th Annual Collaboration, Electronic messaging, Anti-Abuse and Spam Conference

Pages 220 - 226

https://doi.org/10.1145/2030376.2030402

Published: 01 September 2011 Publication History

Abstract

IP reputation is a common technique to address email spam problem and while there are commercial implementations available, the algorithms behind them are confidential. A few open source implementations (gossip, RepuScore, IP-GroupREP, etc.) are available, but few studies compare their commercial counterparts.

For this reason, we have made an empirical comparison of six popular commercial IP reputation databases and three different open-source IP reputation algorithms. We built our own IP reputation database from our email corpus, containing 931,576 email messages from real-time email traffic at an academic ISP. After we processed and classified the corpus, we compared the open-source IP reputation algorithms' results with commercial IP reputation databases by using the Spearman rank correlation coefficient to identify the optimal parameters for open-source algorithms.

The results show lower correlation coefficients when the frequency of emails from a single IP is rising. Open-source algorithms performed sufficiently for IP numbers with more than five and less than 50 emails from a single IP, while (surprisingly) the correlation dropped with a higher number of emails from a single IP. For this reason, we believe there should be some additional fine-tuning of open-source algorithms to make them comparable to their commercial counterparts that have IP reputation scores built from many sensors around the world.

We also compared commercial IP reputation databases and found mixed correlations between them, which raised many questions regarding the algorithms used for building IP reputation scores. The research also identified the problem of finding a good methodology for comparing IP reputation databases.

References

[1]

Academic and research network of slovenia. Available from: http://www.arnes.si/en/.

[2]

amavisd-new. Available from: http://www.ijs.si/software/amavisd/.

[3]

The apache spamassassin project. Available from: http://spamassassin.apache.org/.

[4]

The gossip project. Available from: http://www.sufficiently-advanced.net/ {cited 2010-01-28}.

[5]

Not just another bogus list. Available from: http://www.njabl.org/.

[6]

Spamassassin mass checks settings. Available from: http://svn.apache.org/viewvc/spamassassin/branches/3.2/masses/config.set3?view=markup.

[7]

Spamcop {online}. Available from: http://www.spamcop.net/.

[8]

The spamhaus project. Available from: http://www.spamhaus.org/.

[9]

E. Allman, J. Callas, M. Delany, M. Libbey J. Fenton, and M. Thomas. DomainKeys Identified Mail (DKIM) Signatures. RFC 4871 (Proposed Standard), May 2007. Updated by RFC 5672. Available from: http://www.ietf.org/rfc/rfc4871.txt.

[10]

M. Antonakakis. R. Perdisci, D. Dagon. W. Lee. and N. Feamster. Building a dynamic reputation system for dns. In USENIX Security Symposium, pages 273--290, 2010.

Digital Library

[11]

L. Atkins. Public reputation data. Available from: http://blog.wordtothewise.com/2010/05/public-reputation-data/.

[12]

R. Clayton. On the difficulty of counting spam sources. In Seventh Conference on Email and Anti-Spam, 2010.

[13]

T. Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861--874, 2006.

Digital Library

[14]

J. Golbeck and J. Hendler. Reputation network analysis for email filtering. In In Proc. of the Conference on Email and Anti-Spam (CEAS). Mountain View, 2004.

[15]

E. Harris. The next step in the spam control war: Greylisting, 08 2003. Available from: http://projects.puremagic.com/greylisting/.

[16]

I. S. Inc. Senderbase reputation score overview. Available from: http://www.ironport.com/pdf/ironport_senderbase_reputationscore_overview.pdf.

[17]

S. M. Intelligence. January 2011 intelligence report. Available from: http://www.messagelabs.com/mlireport/MLI_2011_01_January_Final_en-us.pdf {cited 2010-01-28}.

[18]

J. Levine. DNS Blacklists and Whitelists. RFC 5782 (Informational). Feb. 2010. Available from: http://www.ietf.org/rfc/rfc5782.txt.

[19]

J. Makey. Blacklists compared, January 2011. Available from: http://www.sdsc.edu/~jeff/spam/cbc.html.

[20]

D. Schatzmann, M. Burkhart, and T. Spyropoulos. Inferring spammers in the network core. In PAM, pages 229--238, 2009.

Digital Library

[21]

G. Singaraju and B. B. Kang. Repuscore: collaborative reputation management framework for email infrastructure. In Proceedings of the 21st conference on Large Installation System. Administration Conference, pages 19:1--19:9, Berkeley. CA, USA. 2007. USENIX Association.

Digital Library

[22]

S. Sinha, M. Bailey, and F. Jahanian. Shades of grey: On the effectiveness of reputation-based blacklists. In Proceedings of the 3rd International Conference on Malicious and Unwanted Software (MALWARE '08), pages 57--64. Fairfax, Virginia, USA, October 2008.

[23]

J. Snyder. Best practices in using reputation-based anti-spam services for email security. http://www.opusl.com/www/whitepapers-/reputationserviceswp.pdf, February 2009.

[24]

W. Van Wanrooij and A. Pras. Filtering spam from bad neighborhoods. Int. J. Netw. Manag., 20:433--444, November 2010.

Digital Library

[25]

M. Vuk and T. Curk. ROC Curve, Lift Chart and Calibration Plot. Metodološki zvezki, 3(1):89--108. 2006.

[26]

A. G. West, A. J. Aviv, J. Chang, and I. Lee. Spam mitigation using spatio-temporal reputations from blacklist history. In Proceedings of the 26th Annual Computer Security Applications Conference, ACSAC '10, pages 161--170, New York, NY, USA, 2010. ACM.

Digital Library

[27]

M. Wong and W. Schlitt. Sender Policy Framework (SPF) for Authorizing Use of Domains in E-Mail, Version 1. RFC 4408 (Experimental), Apr. 2006. Available from: http://www.ietf.org/rfc/rfc4408.txt.

[28]

M. Xie and H. Wang. A collaboration-based autonomous reputation system for email services. In Proceedings of the 29th conference on Information communications, INFOCOM'10. pages 992--1000, Piscataway, NJ, USA, 2010. IEEE Press.

Digital Library

[29]

H. Zhang, H. Duan, W. Liu, and J. Wu. Ipgrouprep: A novel reputation based system for anti-spam. In Proceedings of the 2009 Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing, UIC-ATC '09, pages 513--518, Washington, DC, USA, 2009. IEEE Computer Society.

Digital Library

[30]

Y. X. Zhiyun Qian, Zhuoqing Mao and F. Yu. On network-level clusters for spam detection. In 17th Annual Network and Distributed System, Symposium 2010, 2010.

[31]

T. Zink. Measuring spam effectiveness -- the flip side of false positives. Available from: http://bit.ly/dpNsy4.

Cited By

Lasantha NTilwari VMaduranga MAbeysekara RChakraborty NSharma D(2024)Validating IP Reputation in Cloud Firewall Systems Using Machine Learning Driven Signature Generation and Detection Techniques2024 IEEE Industrial Electronics and Applications Conference (IEACon)10.1109/IEACon61321.2024.10797246(230-235)Online publication date: 4-Nov-2024
https://doi.org/10.1109/IEACon61321.2024.10797246
Huang YNegrete JWagener JFralick CRodriguez APeterson EWosotowsky A(2022)Graph neural networks and cross-protocol analysis for detecting malicious IP addressesComplex & Intelligent Systems10.1007/s40747-022-00838-y9:4(3857-3869)Online publication date: 14-Sep-2022
https://doi.org/10.1007/s40747-022-00838-y
Lewis JTambaliuc GNarman HYoo W(2020)IP Reputation Analysis of Public Databases and Machine Learning Techniques2020 International Conference on Computing, Networking and Communications (ICNC)10.1109/ICNC47757.2020.9049760(181-186)Online publication date: Feb-2020
https://doi.org/10.1109/ICNC47757.2020.9049760
Show More Cited By

Index Terms

Empirical comparison of IP reputation databases
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Metrics
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
  2. World Wide Web
    1. Online advertising
    2. Web applications
      1. Electronic commerce
      2. Internet communications tools
        Email

Recommendations

Preventing Spam Email by Delivery Limitation in RMX
IDEAS '15: Proceedings of the 19th International Database Engineering & Applications Symposium

On the rule-based email exchange system called RMX, similar to general mailing lists, anyone can send emails by sending to an address unique to RMX. However, there is a security problem that we cannot prevent spam emails and accidentally sending email ...
RFC 5617: DomainKeys Identified Mail (DKIM) Author Domain Signing Practices (ADSP)
How Experts Detect Phishing Scam Emails
CSCW

Phishing scam emails are emails that pretend to be something they are not in order to get the recipient of the email to undertake some action they normally would not. While technical protections against phishing reduce the number of phishing emails ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CEAS '11: Proceedings of the 8th Annual Collaboration, Electronic messaging, Anti-Abuse and Spam Conference

September 2011

230 pages

ISBN:9781450307888

DOI:10.1145/2030376

General Chair:
Vidyasagar Potdar
Curtin University, Australia

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 September 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CEAS '11

CEAS '11: The 8th Annual Collaboration, Electronic messaging, Anti-Abuse and Spam Conference

September 1 - 2, 2011

Perth, Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
239
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lasantha NTilwari VMaduranga MAbeysekara RChakraborty NSharma D(2024)Validating IP Reputation in Cloud Firewall Systems Using Machine Learning Driven Signature Generation and Detection Techniques2024 IEEE Industrial Electronics and Applications Conference (IEACon)10.1109/IEACon61321.2024.10797246(230-235)Online publication date: 4-Nov-2024
https://doi.org/10.1109/IEACon61321.2024.10797246
Huang YNegrete JWagener JFralick CRodriguez APeterson EWosotowsky A(2022)Graph neural networks and cross-protocol analysis for detecting malicious IP addressesComplex & Intelligent Systems10.1007/s40747-022-00838-y9:4(3857-3869)Online publication date: 14-Sep-2022
https://doi.org/10.1007/s40747-022-00838-y
Lewis JTambaliuc GNarman HYoo W(2020)IP Reputation Analysis of Public Databases and Machine Learning Techniques2020 International Conference on Computing, Networking and Communications (ICNC)10.1109/ICNC47757.2020.9049760(181-186)Online publication date: Feb-2020
https://doi.org/10.1109/ICNC47757.2020.9049760
Huang YNegrete JWosotowsky AWagener JPeterson ERodriguez AFralick C(2019)Detect Malicious IP Addresses using Cross-Protocol Analysis2019 IEEE Symposium Series on Computational Intelligence (SSCI)10.1109/SSCI44817.2019.9003003(664-672)Online publication date: Dec-2019
https://doi.org/10.1109/SSCI44817.2019.9003003
Gojmerac IZwickl PKovacs GSteindl C(2015)Large-scale active measurements of DNS entries related to e-mail system security2015 IEEE International Conference on Communications (ICC)10.1109/ICC.2015.7249513(7426-7432)Online publication date: Jun-2015
https://doi.org/10.1109/ICC.2015.7249513

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents