Article

Gray-box extraction of execution graphs for anomaly detection

Authors:

Michael K. Reiter,

Dawn SongAuthors Info & Claims

CCS '04: Proceedings of the 11th ACM conference on Computer and communications security

Pages 318 - 329

https://doi.org/10.1145/1030083.1030126

Published: 25 October 2004 Publication History

Abstract

Many host-based anomaly detection systems monitor a process by observing the system calls it makes, and comparing these calls to a model of behavior for the program that the process should be executing. In this paper we introduce a new model of system call behavior, called an <i>execution graph</i>. The execution graph is the first such model that both requires no static analysis of the program source or binary, and conforms to the control flow graph of the program. When used as the model in an anomaly detection system monitoring system calls, it offers two strong properties: (i) it accepts only system call sequences that are consistent with the control flow graph of the program; (ii) it is maximal given a set of training data, meaning that any extensions to the execution graph could permit some intrusions to go undetected. In this paper, we formalize and prove these claims. We additionally evaluate the performance of our anomaly detection technique.

References

[1]

C. Collberg, C. Thomborson and D. Low. Manufacturing cheap, resilient, and stealthy opaque constructs. In Proceedings of the ACM Symposium on Principles of Programming Languages, January 1998.]]

Digital Library

[2]

H. Feng, J. Giffin, Y. Huang, S. Jha, W. Lee and B. Miller. Formalizing sensitivity in static analysis for intrusion detection. In Proceedings of the 2004 IEEE Symposium on Security and Privacy, May 2004.]]

[3]

H. Feng, O. Kolesnikov, P. Fogla, W. Lee and W. Gong. Anomaly detection using call stack information. In Proceedings of the 2003 IEEE Symposium on Security and Privacy, pages 62--75, May 2003.]]

Digital Library

[4]

S. Forrest, S. Hofmeyr, A. Somayaji, and T. Longstaff. A sense of self for Unix processes. In Proceedings of the 1996 IEEE Symposium on Security and Privacy, pages 120--128, May 1996.]]

Digital Library

[5]

D. Gao, M. K. Reiter and D. Song. On gray-box program tracking for anomaly detection. In Proceedings of the 13th USENIX Security Symposium, pages 103--118, August 2004.]]

Digital Library

[6]

J. Giffin, S. Jha and B. Miller. Detecting manipulated remote call streams. In Proceedings of the 11th USENIX Security Symposium, August 2002.]]

Digital Library

[7]

J. Giffin, S. Jha and B. Miller. Efficient context-sensitive intrusion detection. In Proceeding of Symposium on Network and Distributed System Security, Febuary 2004.]]

[8]

C. Kruegel, D. Mutz, F. Valeur and G. Vigna. On the detection of anomalous system call arguments. In Proceeding of ESORICS 2003, October 2003.]]

[9]

X. Lu. A Linux executable editing library. Master's Thesis, Computer and Information Science, National Unviersity of Singpaore. 1999.]]

[10]

M. Prasad and T. Chiueh. A binary rewriting defense against stack based buffer overflow attacks. In USENIX Annual Technical Conference, General Track, June 2003.]]

[11]

N. Provos. Improving host security with system call policies. In Proceeding of the 12th USENIX Security Symposium, August 2003.]]

Digital Library

[12]

N. Provos, M. Friedl and P. Honeyman. Preventing privilege escalation. In Proceeding of the 12th USENIX Security Symposium, August 2003.]]

Digital Library

[13]

T. Romer, G. Voelker, D. Lee, A. Wolman, W. Wong, H. Levy, B. Bershad and B. Chen. Instrumentation and optimization of Win32/Intel executables using etch. In Proceeding of the USENIX Windows NT workshop, August 1997.]]

Digital Library

[14]

B. Schwarz, S. Debray and G. Andrews. Disassembly of executable code revisited. In Proceeding of Working Conference on Reverse Engineering, pages 45--54, Oct 2002.]]

Digital Library

[15]

R. Sekar, M. Bendre, D. Dhurjati and P. Bollineni. A fast automaton-based method for detecting anomalous program behaviors. In Proceedings of the 2001 IEEE Symposium on Security and Privacy, pages 144--155, May 2001.]]

Digital Library

[16]

K. Tan and R. Maxion. "Why 6?"---Defining the operational limits of stide, an anomaly-based intrusion detector. In Proceedings of the 2002 IEEE Symposium on Security and Privacy, pages 188--201, May 2002.]]

Digital Library

[17]

D. Wagner. Janus: an approach for confinement of untrusted applications. Technical Report CSD-99-1056, Department of Computer Science, University of California at Berkeley, August 1999.]]

Digital Library

[18]

D. Wagner and D. Dean. Intrusion detection via static analysis. In Proceedings of the 2001 IEEE Symposium on Security and Privacy, pages 156--168, May 2001.]]

Digital Library

[19]

R. Wahbe, S. Lucco, T. E. Anderson and S. L. Graham. Efficient software-based fault isolation. In Proceeding of the Symposium on Operating System Principles, 1993.]]

Digital Library

[20]

A. Wespi, M. Dacier and H. Debar. An intrusion-detection system based on the Teiresias pattern-discovery algorithm. In Proceedings of the 1999 European Institute for Computer Anti-Virus Research Conference, 1999.]]

[21]

A. Wespi, M. Dacier and H. Debar. Intrusion detection using variable-length audit trail patterns. In Proceedings of the 2000 Recent Advances in Intrusion Detection, pages 110--129, October 2000.]]

Digital Library

Cited By

Busquim e Silva RArai NBurgareli LParente de Oliveira JSousa Pinto J(2024)Exploring Frama-C Resources by Verifying Space SoftwareGuide to Software Verification with Frama-C10.1007/978-3-031-55608-1_14(583-615)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-55608-1_14
Xie ZPei CLi WJiang HSu LLi JXie GPei DChandra SBlincoe KTonella P(2023)From Point-wise to Group-wise: A Fast and Accurate Microservice Trace Anomaly Detection ApproachProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613861(1739-1749)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613861
Bauman EDuan JHamlen KLin Z(2023)Renewable Just-In-Time Control-Flow IntegrityProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3607199.3607239(580-594)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3607199.3607239
Show More Cited By

Index Terms

Gray-box extraction of execution graphs for anomaly detection
1. Security and privacy
  1. Systems security
    1. Operating systems security

Recommendations

Scalable anomaly detection in graphs

The advantage of graph-based anomaly detection is that the relationships between elements can be analyzed for structural oddities that could represent activities such as fraud, network intrusions, or suspicious associations in a social network. ...
Program Anomaly Detection: Methodology and Practices
CCS '16: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security

This tutorial will present an overview of program anomaly detection, which analyzes normal program behaviors and discovers aberrant executions caused by attacks, misconfigurations, program bugs, and unusual usage patterns. It was first introduced as an ...
Specification-based anomaly detection: a new approach for detecting network intrusions
CCS '02: Proceedings of the 9th ACM conference on Computer and communications security

Unlike signature or misuse based intrusion detection techniques, anomaly detection is capable of detecting novel attacks. However, the use of anomaly detection in practice is hampered by a high rate of false alarms. Specification-based techniques have ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CCS '04: Proceedings of the 11th ACM conference on Computer and communications security

October 2004

376 pages

ISBN:1581139616

DOI:10.1145/1030083

General Chair:
Vijay Atluri
Rutgers University
,
Program Chairs:
Birgit Pfitzmann
IBM Research, Switzerland
,
Patrick McDaniel
AT&T Labs

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CCS04

Sponsor:

CCS04: 11th ACM Conference on Computer and Communications Security 2004

October 25 - 29, 2004

Washington DC, USA

Acceptance Rates

Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '24

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 14 - 18, 2024

Salt Lake City , UT , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

76
Total Citations
View Citations
1,173
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Busquim e Silva RArai NBurgareli LParente de Oliveira JSousa Pinto J(2024)Exploring Frama-C Resources by Verifying Space SoftwareGuide to Software Verification with Frama-C10.1007/978-3-031-55608-1_14(583-615)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-55608-1_14
Xie ZPei CLi WJiang HSu LLi JXie GPei DChandra SBlincoe KTonella P(2023)From Point-wise to Group-wise: A Fast and Accurate Microservice Trace Anomaly Detection ApproachProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613861(1739-1749)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613861
Bauman EDuan JHamlen KLin Z(2023)Renewable Just-In-Time Control-Flow IntegrityProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3607199.3607239(580-594)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3607199.3607239
Yang YZhang M(2023)From Tactics to Techniques: A Systematic Attack Modeling for Advanced Persistent Threats in Industrial Control Systems2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)10.1109/EuroSPW59978.2023.00042(336-344)Online publication date: Jul-2023
https://doi.org/10.1109/EuroSPW59978.2023.00042
Song SSuneja SLe MTak B(2023)On the Value of Sequence-Based System Call Filtering for Container Security2023 IEEE 16th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD60044.2023.00043(296-307)Online publication date: Jul-2023
https://doi.org/10.1109/CLOUD60044.2023.00043
Song SSuneja SLe MTak B(2023)Sequence-based System Call Filtering for Enhanced Container Security, is it beneficial?2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW)10.1109/CCGridW59191.2023.00057(278-280)Online publication date: May-2023
https://doi.org/10.1109/CCGridW59191.2023.00057
Chaturvedi A(2022)Call Graph Evolution Analytics over a Version Series of an Evolving Software SystemProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3559573(1-5)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3559573
Chen S(2022)From Dependability to Security—A Path in the Trustworthy Computing ResearchSystem Dependability and Analytics10.1007/978-3-031-02063-6_4(55-67)Online publication date: 26-Jul-2022
https://doi.org/10.1007/978-3-031-02063-6_4
Xiong CZhu TDong WRuan LYang RChen YCheng YCheng SChen X(2020)CONAN: A Practical Real-time APT Detection System with High Accuracy and EfficiencyIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2020.2971484(1-1)Online publication date: 2020
https://doi.org/10.1109/TDSC.2020.2971484
Hossain MSheikhi SSekar R(2020)Combating Dependence Explosion in Forensic Analysis Using Alternative Tag Propagation Semantics2020 IEEE Symposium on Security and Privacy (SP)10.1109/SP40000.2020.00064(1139-1155)Online publication date: May-2020
https://doi.org/10.1109/SP40000.2020.00064
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents