Article

Evading network anomaly detection systems: formal reasoning and practical techniques

Authors:

Wenke LeeAuthors Info & Claims

CCS '06: Proceedings of the 13th ACM conference on Computer and communications security

Pages 59 - 68

https://doi.org/10.1145/1180405.1180414

Published: 30 October 2006 Publication History

Abstract

Attackers often try to evade an intrusion detection system (IDS) when launching their attacks. There have been several published studies in evasion attacks, some with available tools, in the research community as well as the "hackers'' community. Our recent empirical case study showed that some payload-based network anomaly detection systems can be evaded by a polymorphic blending attack (PBA). The main idea of a PBA is to create each polymorphic instance in such a way that the statistics of attack packet(s) match the normal traffic profile. In this paper, we present a formal framework for the open problem: given an anomaly detection system and an attack, can one automatically generate its PBA instances? We show that in general, generating a PBA that optimally matches the normal traffic profile is a hard problem (NP-complete). However, the problem of finding a PBA can be reduced to the SAT or ILP problems so that solvers available in those domains can be used to find a near-optimal solution. We also present a heuristic (hill-climbing) to find an approximate solution. Our framework can not only expose how the IDS can be exploited by a PBA but also suggest how the IDS can be improved to prevent the PBA. We have experimented with our framework using the PAYL 1-gram and 2-gram anomaly detection system, and the results have validated our framework.

References

[1]

P. Akritidis, E. P. Markatos, M. Polychronakis, and K. Anagnostakis. Stride: Polymorphic sled detection through instruction sequence analysis. In 20th IFIP International Information Security Conference, 2005.

[2]

M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar. Can machine learning be secure? In Proceedings of the ACM Symposium on Information, Computer, and Communication Security (ASIACCS), 2006.

Digital Library

[3]

M. Christodorescu, S. Jha, S. Seshia, D. Song, and R. Bryant. Semantics-aware malware detection. In Proceedings of the IEEE Symposium on Security and Privacy, 2005.

Digital Library

[4]

T. H. Cormen, C. E. Leiserson, and R. L. Rivest. Introduction to algorithms. The MIT Press/McGraw-Hill, 1990.

Digital Library

[5]

T. Detristan, T. Ulenspiegel, Y. Malcom, and M. Underduk. Polymorphic shellcode engine using spectrum analysis. Phrack Issue 0x3d, 2003.

[6]

S. T. Eckmann, G. Vigna, and R. A. Kemmerer. Statl: An attack language for state-based intrusion detection. JOURNAL OF COMPUTER SECURITY, 10:71--104, 2002.

Digital Library

[7]

H. Feng, J. Giffin, Y. Huang, S. Jha, W. Lee, and B. Miller. Formalizing sensitivity in static analysis for intrusion detection. In Proceedings of the IEEE Symposium on Security and Privacy, 2004.

[8]

H. Feng, O. Kolesnikov, P. Fogla, W. Lee, and W. Gong. Anomaly detection using call stack information. In Proceedings of the IEEE Symposium on Security and Privacy, 2003.

Digital Library

[9]

Firew0rker. Windows media services remote command execution exploit. http://www.k-otik.com/exploits/07.01.nsiilog-titbit.cpp.php, 2003.

[10]

P. Fogla, M. Sharif, R. Perdisci, O. M. Kolesnikov, and W. Lee. Polymorphic blending attacks. In 15th USENIX Security Symposium, 2006.

Digital Library

[11]

C. Kaufman, R. Perlman, and M. Speciner. Network security: Private communication in a public world. Prentice Hall, 2002.

Digital Library

[12]

C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna. Automating mimicry attacks using static binary analysis. In 14th Usenix Security Symposium, 2005.

Digital Library

[13]

C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna. Polymorphic worm detection using structural information of executables. In Recent Advances in Intrusion Detection (RAID), 2005.

Digital Library

[14]

C. Kruegel, T. Toth, and E. Kirda. Service specific anomaly detection for network intrusion detection. In Proceedings of the ACM SIGSAC, 2002.

Digital Library

[15]

C. Kruegel and G. Vigna. Anomaly detection of web-based attacks. In Proceedings of the ACM Conference on Computer and Communication Security (ACM CCS), pages 251--261, 2003.

Digital Library

[16]

Ktwo. Admmutate: Shellcode mutation engine. http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz, 2001.

[17]

Z. Liang and R. Sekar. Fast and automated generation of attack signatures: a basis for building self-protecting servers. Proceedings of the 12th ACM Conference on Computer and Communications Security (ACM CCS), pages 213--222, 2005.

Digital Library

[18]

M. Mahoney. Network traffic anomaly detection based on packet bytes. In Proceedings of the ACM SIGSAC, 2003.

Digital Library

[19]

M. Mahoney and P.K. Chan. Learning nonstationary models of normal network traffic for detecting novel attacks.

[20]

J. Newsome, B. Karp, and D. Song. Polygraph: Automatically generating signatures for polymorphic worms. In Proceedings of the IEEE Symposium on Security and Privacy, 2005.

Digital Library

[21]

Martin Roesch. Snort-lightweight intrusion detection for networks. In Proceedings of the 13th USENIX conference on System administration, pages 229--238, 1999.

Digital Library

[22]

S. Rubin, S. Jha, and B. P. Miller. Language-based generation and evaluation of nids signatures. In Proceedings of the IEEE Symposium on Security and Privacy, 2005.

Digital Library

[23]

S. Rubin, S. Jha, and B.P. Miller. Automatic generation and analysis of nids attacks. In Annual Computer Security Applications Conference (ACSAC), 2004.

Digital Library

[24]

R. Sekar, M. Bendre, D. Dhurjati, and P. Bollineni. A fast automaton-based method for detecting anomalous program behaviors. In Proceedings of the IEEE Symposium on Security and Privacy, 2001.

Digital Library

[25]

R. Sekar, A. Gupta, J. Frullo, T. Shanbhag, A. Tiwari, H. Yang, and S. Zhou. Specification-based anomaly detection: A new approach for detecting network intrusions. In Proceedings of the ACM conference on Computer and communications security (ACM CCS), 2002.

Digital Library

[26]

C. Sinz. Towards an optimal cnf encoding of boolean cardinality constraints. In Principles and Practice of Constraint Programming, pages 827--831, 2005.

[27]

P. Szor. Advanced code evolution techniques and computer virus generator kits. The Art of Computer Virus Research and Defense, 2005.

[28]

T. Toth and C. Kruegel. Accurate buffer overflow detection via abstract payload execution. In Recent Advances in Intrusion Detection (RAID), 2002.

Digital Library

[29]

G. Vigna, W. Robertson, and D. Balzarotti. Testing network-based intrusion detection signatures using mutant exploits. In Proceedings of the ACM Conference on Computer and Communication Security (ACM CCS), pages 21--30, 2004.

Digital Library

[30]

D. Wagner and P. Soto. Mimicry attacks on host-based intrusion detection systems. In Proceedings of the ACM Conference on Computer and Communication Security (ACM CCS), 2002.

Digital Library

[31]

K. Wang and S. Stolfo. Anomalous payload-based network intrusion detection. In Recent Advances in Intrusion Detection (RAID), 2004.

[32]

K. Wang and S. Stolfo. Anomalous payload-based worm detection and signature generation. In Recent Advances in Intrusion Detection (RAID), 2005.

Digital Library

[33]

T. Yetiser. Polymorphic viruses: Implementation, detection, and protection. Technical Report, VDS Advanced Research Group, 1993.

Cited By

Sen JDasgupta S(2023)Adversarial Attacks on Image Classification Models: FGSM and Patch Attacks and Their ImpactInformation Security and Privacy in the Digital World - Some Selected Topics10.5772/intechopen.112442Online publication date: 27-Sep-2023
https://doi.org/10.5772/intechopen.112442
Javaheripi MSamragh MRouhani BJavidi TKoushanfar F(2021)CuRTAIL: ChaRacterizing and Thwarting AdversarIal Deep LearningIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2020.302419118:2(736-752)Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1109/TDSC.2020.3024191
Han DWang ZZhong YChen WYang JLu SShi XYin X(2021)Evaluating and Improving Adversarial Robustness of Machine Learning-Based Network Intrusion DetectorsIEEE Journal on Selected Areas in Communications10.1109/JSAC.2021.308724239:8(2632-2647)Online publication date: Aug-2021
https://doi.org/10.1109/JSAC.2021.3087242
Show More Cited By

Index Terms

Evading network anomaly detection systems: formal reasoning and practical techniques
1. Security and privacy
  1. Network security

Recommendations

Unknown Attacks Detection Using Feature Extraction from Anomaly-Based IDS Alerts
SAINT '12: Proceedings of the 2012 IEEE/IPSJ 12th International Symposium on Applications and the Internet

Intrusion Detection Systems (IDSs) play an important role detecting various kinds of attacks and defend our computer systems from them. There are basically two main types of detection techniques: signature-based and anomaly-based. A signature-based IDS ...
Rule generalisation in intrusion detection systems using SNORT

Intrusion Detection Systems (IDSs) provide an important layer of security for computer systems and networks. An IDS's responsibility is to detect suspicious or unacceptable system and network activity and to alert a systems administrator to this ...
Mimicry attacks on host-based intrusion detection systems
CCS '02: Proceedings of the 9th ACM conference on Computer and communications security

We examine several host-based anomaly detection systems and study their security against evasion attacks. First, we introduce the notion of a mimicry attack, which allows a sophisticated attacker to cloak their intrusion to avoid detection by the IDS. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CCS '06: Proceedings of the 13th ACM conference on Computer and communications security

October 2006

434 pages

ISBN:1595935185

DOI:10.1145/1180405

General Chair:
Ari Juels
RSA Laboratories
,
Program Chairs:
Rebecca Wright
Stevens Institute of Technology
,
Sabrina De Capitani di Vimercati
University of Milan, Italy

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CCS06

Sponsor:

CCS06: 13th ACM Conference on Computer and Communications Security 2006

October 30 - November 3, 2006

Virginia, Alexandria, USA

Acceptance Rates

Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '24

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 14 - 18, 2024

Salt Lake City , UT , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

87
Total Citations
View Citations
2,302
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)4

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sen JDasgupta S(2023)Adversarial Attacks on Image Classification Models: FGSM and Patch Attacks and Their ImpactInformation Security and Privacy in the Digital World - Some Selected Topics10.5772/intechopen.112442Online publication date: 27-Sep-2023
https://doi.org/10.5772/intechopen.112442
Javaheripi MSamragh MRouhani BJavidi TKoushanfar F(2021)CuRTAIL: ChaRacterizing and Thwarting AdversarIal Deep LearningIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2020.302419118:2(736-752)Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1109/TDSC.2020.3024191
Han DWang ZZhong YChen WYang JLu SShi XYin X(2021)Evaluating and Improving Adversarial Robustness of Machine Learning-Based Network Intrusion DetectorsIEEE Journal on Selected Areas in Communications10.1109/JSAC.2021.308724239:8(2632-2647)Online publication date: Aug-2021
https://doi.org/10.1109/JSAC.2021.3087242
Wang QFang LZhu ZHuang J(2021)Detection Algorithm of the Mimicry Attack based on Variational Auto-Encoder2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W)10.1109/DSN-W52860.2021.00029(114-120)Online publication date: Jun-2021
https://doi.org/10.1109/DSN-W52860.2021.00029
Kumar SGupta SArora S(2021)Research Trends in Network-Based Intrusion Detection Systems: A ReviewIEEE Access10.1109/ACCESS.2021.31297759(157761-157779)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3129775
Dedousis PStergiopoulos GGritzalis D(2021)An Improved Bit Masking Technique to Enhance Covert Channel Attacks in Everyday IT SystemsE-Business and Telecommunications10.1007/978-3-030-90428-9_1(1-23)Online publication date: 31-Oct-2021
https://doi.org/10.1007/978-3-030-90428-9_1
Ugur A(2021)Manipulator: A Novel Collusion Attack on Trust Management Systems in Social IoTSoftware Engineering and Algorithms10.1007/978-3-030-77442-4_49(578-592)Online publication date: 20-Jul-2021
https://doi.org/10.1007/978-3-030-77442-4_49
Pierazzi FPendlebury FCortellazzi JCavallaro L(2020)Intriguing Properties of Adversarial ML Attacks in the Problem Space2020 IEEE Symposium on Security and Privacy (SP)10.1109/SP40000.2020.00073(1332-1349)Online publication date: May-2020
https://doi.org/10.1109/SP40000.2020.00073
Singh PTapaswi SGupta S(2020)Malware Detection in PDF and Office Documents: A surveyInformation Security Journal: A Global Perspective10.1080/19393555.2020.172374729:3(134-153)Online publication date: 13-Feb-2020
https://doi.org/10.1080/19393555.2020.1723747
Tong LLi BHajaj CXiao CZhang NVorobeychik YHeninger NTraynor P(2019)Improving robustness of ML classifiers against realizable evasion attacks using conserved featuresProceedings of the 28th USENIX Conference on Security Symposium10.5555/3361338.3361359(285-302)Online publication date: 14-Aug-2019
https://dl.acm.org/doi/10.5555/3361338.3361359
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents