research-article

Open problems in the security of learning

Authors:

Peter L. Bartlett,

Fuching Jack Chi,

Anthony D. Joseph,

Benjamin I.P. Rubinstein,

J. D. TygarAuthors Info & Claims

AISec '08: Proceedings of the 1st ACM workshop on Workshop on AISec

Pages 19 - 26

https://doi.org/10.1145/1456377.1456382

Published: 27 October 2008 Publication History

Abstract

Machine learning has become a valuable tool for detecting and preventing malicious activity. However, as more applications employ machine learning techniques in adversarial decision-making situations, increasingly powerful attacks become possible against machine learning systems. In this paper, we present three broad research directions towards the end of developing truly secure learning. First, we suggest that finding bounds on adversarial influence is important to understand the limits of what an attacker can and cannot do to a learning system. Second, we investigate the value of adversarial capabilities-the success of an attack depends largely on what types of information and influence the attacker has. Finally, we propose directions in technologies for secure learning and suggest lines of investigation into secure techniques for learning in adversarial environments. We intend this paper to foster discussion about the security of machine learning, and we believe that the research directions we propose represent the most important directions to pursue in the quest for secure learning.

References

[1]

Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning. Technical Report UCB/EECS-2008-43, EECS Department, University of California, Berkeley, April 2008.

[2]

Marco Barreno, Blaine Nelson, Russell Sears, Anthony D. Joseph, and J. D. Tygar. Can machine learning be secure? In Proceedings of the ACM Symposium on InformAtion, Computer, and Communications Security (ASIACCS'06), March 2006.

Digital Library

[3]

Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.

Digital Library

[4]

Simon P. Chung and Aloysius K. Mok. Allergy attack against automatic signature generation. In Recent Advances in Intrusion Detection (RAID), pages 61--80, 2006.

Digital Library

[5]

Simon P. Chung and Aloysius K. Mok. Advanced allergy attacks: Does a corpus really help? In Recent Advances in Intrusion Detection (RAID), pages 236--255, 2007.

Digital Library

[6]

Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, and Deepak Verma. Adversarial classification. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 99--108, Seattle, WA, 2004. ACM Press.

Digital Library

[7]

Prahlad Fogla and Wenke Lee. Evading network anomaly detection systems: Formal reasoning and practical techniques. In Proceedings of the ACM Conference on Computer and Communications Security (CCS), pages 59--68, 2006.

Digital Library

[8]

Jason Franklin, Vern Paxson, Adrian Perrig, and Stefan Savage. An inquiry into the nature and causes of the wealth of internet miscreants. In Proceedings of the ACM Conference on Computer and Communications Security (CCS), 2007.

Digital Library

[9]

Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119--139, 1997.

Digital Library

[10]

Frank R. Hampel, Elvezio M. Ronchetti, Peter J. Rousseeuw, and Werner A. Stahel. Robust Statistics: The Approach Based on Influence Functions. Probability and Mathematical Statistics. John Wiley and Sons, 1986.

[11]

Peter J. Huber. Robust Statistics. John Wiley and Sons, 1981.

[12]

Michael Kearns and Ming Li. Learning in the presence of malicious errors. SIAM Journal on Computing, 22:807--837, 1993.

Digital Library

[13]

Anukool Lakhina, Mark Crovella, and Christophe Diot. Diagnosing network--wide traffic anomalies. In Proc. SIGCOMM '04, pages 219--230, 2004.

Digital Library

[14]

Zhichun Li, Manan Sanghi, Yan Chen, Ming-Yang Kao, and Brian Chavez. Hamsa: fast signature generation for zero-day polymorphic worms with provable attack resilience. In IEEE Symposium on Security and Privacy, 2006.

Digital Library

[15]

Daniel Lowd and Christopher Meek. Adversarial learning. In Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 641--647, 2005.

Digital Library

[16]

Daniel Lowd and Christopher Meek. Good word attacks on statistical spam filters. In Proceedings of the Second Conference on Email and Anti-Spam (CEAS), 2005.

[17]

Markos Markou and Sameer Singh. Novelty detection: a review--part 1: statistical approaches. Signal Processing, 83(12):2481--2497, December 2003.

Digital Library

[18]

Ricardo A. Maronna, Douglas R. Martin, and Victor J. Yohai. Robust Statistics: Theory and Methods. John Wiley and Sons, New York, 2006.

[19]

Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D. Joseph, Benjamin I. P. Rubinstein, Udam Saini, Charles Sutton, J. D. Tygar, and Kai Xia. Exploiting machine learning to subvert your spam filter. In Proceedings of the First Workshop on Large-scale Exploits and Emerging Threats (LEET), 2008.

Digital Library

[20]

Blaine Nelson and Anthony D. Joseph. Bounding an attack's complexity for a simple learning model. In Proceedings of the First Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML), 2006.

[21]

James Newsome, Brad Karp, and Dawn Song. Polygraph: Automatically generating signatures for polymorphic worms. In Proceedings of the IEEE Symposium on Security and Privacy, pages 226--241, May 2005.

Digital Library

[22]

James Newsome, Brad Karp, and Dawn Song. Paragraph: Thwarting signature learning by training maliciously. In Proceedings of the 9th International Symposium on Recent Advances in Intrusion Detection (RAID 2006), September 2006.

Digital Library

[23]

PhishTank. http://www.phishtank.com.

[24]

Gary Robinson. A statistical approach to the spam problem. Linux Journal, March 2003.

Digital Library

[25]

Benjamin I. P. Rubinstein, Blaine Nelson, Ling Huang, Anthony D. Joseph, Shing-hon Lau, Nina Taft, and J. D. Tygar. Compromising PCA-based anomaly detectors for network-wide traffic. Technical report UCB/EECS-2008-73, UC Berkeley, May 2008.

[26]

Robert E. Schapire. A brief introduction to boosting. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI '99), pages 1401--1406, 1999.

Digital Library

[27]

Kymie M. C. Tan, Kevin S. Killourhy, and Roy A. Maxion. Undermining an anomaly-based intrusion detection system using common exploits. In Recent Advances in Intrusion Detection (RAID), pages 54--73, 2002.

Digital Library

[28]

Vladimir N. Vapnik and Alexey Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16(2):264--280, 1971.

[29]

Shobha Venkataraman, Avrim Blum, and Dawn Song. Limits of learning-based signature generation with adversaries. In Proceedings of the 15th Annual Network & Distributed System Security Symposium, 2008.

[30]

Gregory L. Wittel and S. Felix Wu. On attacking statistical spam filters. In Proceedings of the First Conference on Email and Anti-Spam (CEAS), 2004.

Cited By

Akın Özdemir N(2024)En Çok Turist Çeken 30 Ülkede Turizm, GSYİH ve Yenilenebilir Enerjinin CO2 Emisyonları Üzerindeki Etkisinin AraştırılmasıAnemon Muş Alparslan Üniversitesi Sosyal Bilimler Dergisi10.18506/anemon.147956112:2(659-672)Online publication date: 30-Aug-2024
https://doi.org/10.18506/anemon.1479561
Correia MRodrigues L(2023)Security and PrivacyMultidisciplinary Perspectives on Artificial Intelligence and the Law10.1007/978-3-031-41264-6_5(81-101)Online publication date: 27-Dec-2023
https://doi.org/10.1007/978-3-031-41264-6_5
Sagar RJhaveri RBorrego C(2020)Applications in Security and Evasions in Machine Learning: A SurveyElectronics10.3390/electronics90100979:1(97)Online publication date: 3-Jan-2020
https://doi.org/10.3390/electronics9010097
Show More Cited By

Index Terms

Open problems in the security of learning

Recommendations

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

In recent years, machine learning algorithms, and more specifically deep learning algorithms, have been widely used in many fields, including cyber security. However, machine learning systems are vulnerable to adversarial attacks, and this limits the ...
Adversarial machine learning
AISec '11: Proceedings of the 4th ACM workshop on Security and artificial intelligence

In this paper (expanded from an invited talk at AISEC 2010), we discuss an emerging field of study: adversarial machine learning---the study of effective machine learning techniques against an adversarial opponent. In this paper, we: give a taxonomy for ...
Can machine learning be secure?
ASIACCS '06: Proceedings of the 2006 ACM Symposium on Information, computer and communications security

Machine learning systems offer unparalled flexibility in dealing with evolving input in a variety of applications, such as intrusion detection systems and spam e-mail filtering. However, machine learning algorithms themselves can be a target of attack ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AISec '08: Proceedings of the 1st ACM workshop on Workshop on AISec

October 2008

84 pages

ISBN:9781605582917

DOI:10.1145/1456377

Program Chairs:
Dirk Balfanz
Google, USA
,
Jessica Staddon
PARC, USA

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CCS08

Sponsor:

CCS08: 15th ACM Conference on Computer and Communications Security 2008

October 27, 2008

Virginia, Alexandria, USA

Acceptance Rates

AISec '08 Paper Acceptance Rate 9 of 20 submissions, 45%;

Overall Acceptance Rate 94 of 231 submissions, 41%

Upcoming Conference

CCS '25

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 13 - 17, 2025

Taipei , Taiwan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
858
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Akın Özdemir N(2024)En Çok Turist Çeken 30 Ülkede Turizm, GSYİH ve Yenilenebilir Enerjinin CO2 Emisyonları Üzerindeki Etkisinin AraştırılmasıAnemon Muş Alparslan Üniversitesi Sosyal Bilimler Dergisi10.18506/anemon.147956112:2(659-672)Online publication date: 30-Aug-2024
https://doi.org/10.18506/anemon.1479561
Correia MRodrigues L(2023)Security and PrivacyMultidisciplinary Perspectives on Artificial Intelligence and the Law10.1007/978-3-031-41264-6_5(81-101)Online publication date: 27-Dec-2023
https://doi.org/10.1007/978-3-031-41264-6_5
Sagar RJhaveri RBorrego C(2020)Applications in Security and Evasions in Machine Learning: A SurveyElectronics10.3390/electronics90100979:1(97)Online publication date: 3-Jan-2020
https://doi.org/10.3390/electronics9010097
Branco P(2020)Exploring the Impact of Resampling Methods for Malware Detection2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378405(3961-3968)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378405
Li BVorobeychik Y(2018)Evasion-Robust Classification on Binary DomainsACM Transactions on Knowledge Discovery from Data10.1145/318628212:4(1-32)Online publication date: 8-Jun-2018
https://dl.acm.org/doi/10.1145/3186282
Almukaynizi MNunes EDharaiya KSenguttuvan MShakarian JShakarian P(2018)Patch Before Exploited: An Approach to Identify Targeted Software VulnerabilitiesAI in Cybersecurity10.1007/978-3-319-98842-9_4(81-113)Online publication date: 18-Sep-2018
https://doi.org/10.1007/978-3-319-98842-9_4
Zhou YKantarcioglu MXi B(2018)A survey of game theoretic approach for adversarial machine learningWIREs Data Mining and Knowledge Discovery10.1002/widm.12599:3Online publication date: 30-Apr-2018
https://doi.org/10.1002/widm.1259
Gardiner JNagaraja S(2016)On the Security of Machine Learning in Malware C&C DetectionACM Computing Surveys10.1145/300381649:3(1-39)Online publication date: 13-Dec-2016
https://dl.acm.org/doi/10.1145/3003816
Sabottke CSuciu ODumitraş T(2015)Vulnerability disclosure in the age of social mediaProceedings of the 24th USENIX Conference on Security Symposium10.5555/2831143.2831209(1041-1056)Online publication date: 12-Aug-2015
https://dl.acm.org/doi/10.5555/2831143.2831209
Neville SElgamal MNikdel Z(2015)Robust adversarial learning and invariant measures2015 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM)10.1109/PACRIM.2015.7334893(529-535)Online publication date: Aug-2015
https://doi.org/10.1109/PACRIM.2015.7334893
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten