research-article

Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation

Authors:

Nathan Dautenhahn,

Theodoros Kasampalis,

Vikram AdveAuthors Info & Claims

ACM SIGPLAN Notices, Volume 50, Issue 4

Pages 191 - 206

https://doi.org/10.1145/2775054.2694386

Published: 14 March 2015 Publication History

Abstract

Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a small isolated kernel within a traditional monolithic kernel. The "nested kernel" interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement. We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware while allowing the entire operating system, including untrusted components, to operate at the highest hardware privilege level by write-protecting MMU translations and de-privileging the untrusted part of the kernel. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks. We also demonstrate that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels by introducing write-mediation and write-logging services to protect critical system data structures. Performance of the nested kernel prototype shows modest overheads: <1% average for Apache and 2.7% for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.

References

[1]

AMD64 architecture programmers manual volume 2: System programming. Manual, Advancd Micro Devices, 2006.

[2]

Intel 64 and IA-32 architectures software developers manual. Manual 325384-051US, Intel, June 2014.

[3]

M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, and M. Young. Mach: A new kernel foundation for UNIX development. In Proceedings of the USENIX Annual Technical Conference, USENIX ATC'10, pages 93--112, Altanta, GA, USA, 1986. USENIX Association.

[4]

M. Aiken, M. Fhndrich, C. Hawblitzel, G. Hunt, and J. Larus. Deconstructing process isolation. In Proceedings of the 2006 Workshop on Memory System Performance and Correctness, MSPC '06, pages 1--10, New York, NY, USA, 2006. ACM.

Digital Library

[5]

argp and Karl. Exploiting UMA, FreeBSD's kernel memory allocator.:: Phrack Magazine ::., 0x0d(0x42), Nov. 2009.

[6]

S. Bahram, X. Jiang, Z. Wang, M. Grace, J. Li, D. Srinivasan, J. Rhee, and D. Xu. DKSM: Subverting virtual machine introspection for fun and profit. In Proceedings of the 2010 29th IEEE Symposium on Reliable Distributed Systems, SRDS '10, pages 82--91, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

[7]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, pages 164--177, New York, NY, USA, 2003. ACM.

Digital Library

[8]

A. Belay, A. Bittau, A. Mashtizadeh, D. Terei, D. Mazires, and C. Kozyrakis. Dune: Safe user-level access to privileged CPU features. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI'12, pages 335--348, Berkeley, CA, USA, 2012. USENIX Association.

Digital Library

[9]

B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer, M. E. Fiuczynski, D. Becker, C. Chambers, and S. Eggers. Extensibility safety and performance in the SPIN operating system. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 267--283, New York, NY, USA, 1995. ACM.

Digital Library

[10]

M. Castro, M. Costa, J.-P. Martin, M. Peinado, P. Akritidis, A. Donnelly, P. Barham, and R. Black. Fast byte-granularity software fault isolation. In Proceedings of the ACM SIGOPS 22nd symposium on Operating Systems Principles, SOSP '09, pages 45--58, New York, NY, USA, 2009. ACM.

Digital Library

[11]

S. Chen, J. Xu, E. C. Sezer, P. Gauriar, and R. K. Iyer. Non-control-data attacks are realistic threats. In Proceedings of the 14th Conference on USENIX Security Symposium - Volume 14, SSYM'05, pages 12--12, Berkeley, CA, USA, 2005. USENIX Association.

Digital Library

[12]

A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. An empirical study of operating systems errors. In Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, SOSP '01, pages 73--88, New York, NY, USA, 2001. ACM.

Digital Library

[13]

J. Criswell, N. Dautenhahn, and V. Adve. KCoFI: Complete control-flow integrity for commodity operating system kernels. In Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP '14, pages 292--307, Washington, DC, USA, 2014. IEEE Computer Society.

Digital Library

[14]

J. Criswell, N. Dautenhahn, and V. Adve. Virtual ghost: Protecting applications from hostile operating systems. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '14, pages 81--96, New York, NY, USA, 2014. ACM.

Digital Library

[15]

J. Criswell, N. Geoffray, and V. Adve. Memory safety for low-level software/hardware interactions. In Proceedings of the 18th Conference on USENIX Security Symposium, SSYM'09, pages 83--100, Berkeley, CA, USA, 2009. USENIX Association.

Digital Library

[16]

J. Criswell, A. Lenharth, D. Dhurjati, and V. Adve. Secure virtual architecture: A safe execution environment for commodity operating systems. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP '07, pages 351--366, New York, NY, USA, 2007. ACM.

Digital Library

[17]

D. R. Engler, M. F. Kaashoek, and J. O'Toole, Jr. Exokernel: An operating system architecture for application-level resource management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 251--266, New York, NY, USA, 1995. ACM.

Digital Library

[18]

U. Erlingsson, M. Abadi, M. Vrable, M. Budiu, and G. C. Necula. XFI: Software guards for system address spaces. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, OSDI '06, pages 75--88, Berkeley, CA, USA, 2006. USENIX Association.

Digital Library

[19]

T. Garfinkel and M. Rosenblum. A virtual machine introspection based architecture for intrusion detection. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2003, San Diego, California, USA. The Internet Society, 2003.

[20]

O. S. Hofmann, S. Kim, A. M. Dunn, M. Z. Lee, and E. Witchel. InkTag: Secure applications on an untrusted operating system. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 265--278, New York, NY, USA, 2013. ACM.

Digital Library

[21]

S. A. Hofmeyr, S. Forrest, and A. Somayaji. Intrusion detection using sequences of system calls. J. Comput. Secur., 6(3):151--180, Aug. 1998.

[22]

N. Honarmand, N. Dautenhahn, J. Torrellas, S. T. King, G. Pokam, and C. Pereira. Cyrus: Unintrusive application-level record-replay for replay parallelism. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 193--206, New York, NY, USA, 2013. ACM.

Digital Library

[23]

N. Honarmand and J. Torrellas. Replay debugging: Leveraging record and replay for program debugging. In Proceeding of the 41st Annual International Symposium on Computer Architecuture, ISCA '14, pages 445--456, Piscataway, NJ, USA, 2014. IEEE Press.

Digital Library

[24]

B. Jain, M. B. Baig, D. Zhang, D. E. Porter, and R. Sion. SoK: Introspections on trust and the semantic gap. In Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP '14, pages 605--620, Washington, DC, USA, 2014. IEEE Computer Society.

Digital Library

[25]

V. P. Kemerlis, M. Polychronakis, and A. D. Keromytis. ret2dir: Rethinking kernel isolation. In Proceedings of the 23rd USENIX Conference on Security Symposium, SEC'14, pages 957--972, Berkeley, CA, USA, 2014. USENIX Association.

Digital Library

[26]

S. T. King and P. M. Chen. Backtracking intrusions. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, pages 223--236, New York, NY, USA, 2003. ACM.

Digital Library

[27]

J. Kong. Designing BSD Rootkits. No Starch Press, San Francisco, CA, USA, 2007.

Digital Library

[28]

J. Liedtke. On micro-kernel construction. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 237--250, New York, NY, USA, 1995. ACM.

Digital Library

[29]

Y. Mao, H. Chen, D. Zhou, X. Wang, N. Zeldovich, and M. F. Kaashoek. Software fault isolation with API integrity and multi-principal modules. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP '11, pages 115--128, New York, NY, USA, 2011. ACM.

Digital Library

[30]

L. McVoy and C. Staelin. lmbench: Portable tools for performance analysis. In Proceedings of the 1996 Annual Conference on USENIX Annual Technical Conference, ATEC '96, pages 23--23, Berkeley, CA, USA, 1996. USENIX Association.

Digital Library

[31]

Microsoft. Kernel patch protection: frequently asked questions (windows drivers), 2007.

[32]

P. Montesinos, M. Hicks, S. T. King, and J. Torrellas. Capo: A software-hardware interface for practical deterministic multiprocessor replay. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIV, pages 73-- 84, New York, NY, USA, 2009. ACM.

Digital Library

[33]

D. Mutz, F. Valeur, G. Vigna, and C. Kruegel. Anomalous system call detection. ACM Trans. Inf. Syst. Secur., 9(1):61--93, Feb. 2006.

Digital Library

[34]

E. I. Organick. The Multics System: An Examination of Its Structure. MIT Press, Cambridge, MA, USA, 1972.

Digital Library

[35]

B. D. Payne, M. Carbone, M. Sharif, and W. Lee. Lares: An architecture for secure active monitoring using virtualization. In Proceedings of the 2008 IEEE Symposium on Security and Privacy, SP '08, pages 233--247, Washington, DC, USA, 2008. IEEE Computer Society.

Digital Library

[36]

G. Pokam, K. Danne, C. Pereira, R. Kassa, T. Kranich, S. Hu, J. Gottschlich, N. Honarmand, N. Dautenhahn, S. T. King, and J. Torrellas. QuickRec: Prototyping an intel architecture extension for record and replay of multithreaded programs. In Proceedings of the 40th Annual International Symposium on Computer Architecture, ISCA '13, pages 643--654, New York, NY, USA, 2013. ACM.

Digital Library

[37]

C. Ries. Defeating windows personal firewalls: Filtering methodologies, attacks, and defenses. Technical report, 2005.

[38]

R. Riley, X. Jiang, and D. Xu. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection, RAID '08, pages 1--20, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[39]

J. M. Rushby. Design and verification of secure systems. In Proceedings of the Eighth ACM Symposium on Operating Systems Principles, SOSP '81, pages 12--21, New York, NY, USA, 1981. ACM.

Digital Library

[40]

J. H. Saltzer. Protection and the control of information sharing in multics. Commun. ACM, 17(7):388--402, July 1974.

Digital Library

[41]

J. H. Saltzer and M. D. Schroeder. The protection of information in computer systems. Proceedings of the IEEE, 63(9):1278--1308, 1975.

[42]

T. Saulpaugh and C. A. Mirho. Inside the JavaOS operating system. Addison-Wesley Reading, 1999.

[43]

A. Seshadri, M. Luk, N. Qu, and A. Perrig. SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP '07, pages 335--350, New York, NY, USA, 2007. ACM.

Digital Library

[44]

M. I. Sharif, W. Lee, W. Cui, and A. Lanzi. Secure in-VM monitoring using hardware virtualization. In Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS '09, pages 477--487, New York, NY, USA, 2009. ACM.

Digital Library

[45]

G. E. Suh, D. Clarke, B. Gassend, M. van Dijk, and S. Devadas. AEGIS: Architecture for tamper-evident and tamper-resistant processing. In Proceedings of the 17th Annual International Conference on Supercomputing, ICS '03, pages 160--171, New York, NY, USA, 2003. ACM.

Digital Library

[46]

M. M. Swift, B. N. Bershad, and H. M. Levy. Improving the reliability of commodity operating systems. ACM Trans. Comput. Syst., 23(1):77--110, Feb. 2005.

Digital Library

[47]

A. Tereshkin. Rootkits: Attacking personal firewalls. In Proceedings of the Black Hat USA 2006 Conference, 2006.

[48]

I. Unified EFI. Unified extensible firmware interface specification: Version 2.2d, November 2010.

[49]

D. Wagner and P. Soto. Mimicry attacks on host-based intrusion detection systems. In Proceedings of the 9th ACM Conference on Computer and Communications Security, CCS '02, pages 255--264, New York, NY, USA, 2002. ACM.

Digital Library

[50]

Z. Wang and X. Jiang. HyperSafe: A lightweight approach to provide lifetime hypervisor control-flow integrity. In Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP '10, pages 380--395, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

[51]

Z. Wang, X. Jiang, W. Cui, and P. Ning. Countering kernel rootkits with lightweight hook protection. In Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS '09, pages 545--554, New York, NY, USA, 2009. ACM.

Digital Library

[52]

C. Warrender, S. Forrest, and B. A. Pearlmutter. Detecting intrusions using system calls: Alternative data models. In 1999 IEEE Symposium on Security and Privacy, SP '99, pages 133--145, Oakland, California, USA, May 1999. IEEE Computer Society.

[53]

D. Wheeler. SLOCCount, 2015. http://www.dwheeler.com/sloccount/.

[54]

C. Wright. Para-virtualization interfaces, 2006. http://lwn.net/Articles/194340.

[55]

X. Xiong and P. Liu. SILVER: Fine-grained and transparent protection domain primitives in commodity OS kernel. In S. J. Stolfo, A. Stavrou, and C. V. Wright, editors, Research in Attacks, Intrusions, and Defenses, number 8145 in Lecture Notes in Computer Science, pages 103--122. Springer Berlin Heidelberg, Jan. 2013.

Digital Library

[56]

M. Xu, X. Jiang, R. Sandhu, and X. Zhang. Towards a VMM-based usage control framework for OS kernel integrity protection. In Proceedings of the 12th ACM Symposium on Access Control Models and Technologies, SACMAT '07, pages 71--80, New York, NY, USA, 2007. ACM.

Digital Library

Cited By

Jiang NZhou QJia XChen JHuang QDu H(2024)LightArmor: A Lightweight Trusted Operating System Isolation Approach for Mobile SystemsICT Systems Security and Privacy Protection10.1007/978-3-031-65175-5_15(206-220)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-65175-5_15
Kuzuno HYamauchi T(2023)Identification of Vulnerable Kernel Code Using Kernel Tracing MechanismJournal of Information Processing10.2197/ipsjjip.31.78831(788-801)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.788
Kuzuno HYamauchi T(2022)Prevention of Kernel Memory Corruption Using Kernel Page Restriction MechanismJournal of Information Processing10.2197/ipsjjip.30.56330(563-576)Online publication date: 2022
https://doi.org/10.2197/ipsjjip.30.563
Show More Cited By

Index Terms

Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation
1. Security and privacy
  1. Systems security
    1. Operating systems security
2. Software and its engineering
  1. Software creation and management
    1. Designing software
  2. Software organization and properties
    1. Contextual software domains
      1. Operating systems

Recommendations

Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation
ASPLOS '15: Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems

Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a ...
Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation
ASPLOS'15

Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a ...
Fast Intra-kernel Isolation and Security with IskiOS
RAID '21: Proceedings of the 24th International Symposium on Research in Attacks, Intrusions and Defenses

The kernels of operating systems such as Windows, Linux, and MacOS are vulnerable to control-flow hijacking. Defenses exist, but many require efficient intra-address-space isolation. Execute-only memory, for example, requires read protection on code ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 50, Issue 4

ASPLOS '15

April 2015

676 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/2775054

Editor:
Andy Gill
University of Kansas, Lawrence, KS

Issue’s Table of Contents

ASPLOS '15: Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems
March 2015
720 pages
ISBN:9781450328357
DOI:10.1145/2694344
General Chairs:
Ozcan Ozturk
Bilkent University, Turkey
,
Kemal Ebcioglu
Global Supercomputing, USA
,
Program Chair:
Sandhya Dwarkadas
University of Rochester, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2015

Published in SIGPLAN Volume 50, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

99
Total Citations
View Citations
1,804
Total Downloads

Downloads (Last 12 months)150
Downloads (Last 6 weeks)18

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jiang NZhou QJia XChen JHuang QDu H(2024)LightArmor: A Lightweight Trusted Operating System Isolation Approach for Mobile SystemsICT Systems Security and Privacy Protection10.1007/978-3-031-65175-5_15(206-220)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-65175-5_15
Kuzuno HYamauchi T(2023)Identification of Vulnerable Kernel Code Using Kernel Tracing MechanismJournal of Information Processing10.2197/ipsjjip.31.78831(788-801)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.788
Kuzuno HYamauchi T(2022)Prevention of Kernel Memory Corruption Using Kernel Page Restriction MechanismJournal of Information Processing10.2197/ipsjjip.30.56330(563-576)Online publication date: 2022
https://doi.org/10.2197/ipsjjip.30.563
Gu JLi HXia YChen HQin CHe Z(2022)Unified Enclave Abstraction and Secure Enclave Migration on Heterogeneous Security ArchitecturesJournal of Computer Science and Technology10.1007/s11390-021-1083-837:2(468-486)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1007/s11390-021-1083-8
Sartakov VVilanova LPietzuch PSherwood TBerger EKozyrakis C(2021)CubicleOS: a library OS with software componentisation for practical isolationProceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3445814.3446731(546-558)Online publication date: 17-Apr-2021
https://doi.org/10.1145/3445814.3446731
Hua ZYu YGu JXia YChen HZang B(2021)TZ-Container: protecting container from untrusted OS with ARM TrustZoneScience China Information Sciences10.1007/s11432-019-2707-664:9Online publication date: 19-Aug-2021
https://doi.org/10.1007/s11432-019-2707-6
Gu JWu XZhu BXia YZang BGuan HChen H(2020)Enclavisor: A Hardware-software Co-design for Enclaves on Untrusted CloudIEEE Transactions on Computers10.1109/TC.2020.3019704(1-1)Online publication date: 2020
https://doi.org/10.1109/TC.2020.3019704
Proskurin SMomeu MGhavamnia SKemerlis VPolychronakis M(2020)xMP: Selective Memory Protection for Kernel and User Space2020 IEEE Symposium on Security and Privacy (SP)10.1109/SP40000.2020.00041(563-577)Online publication date: May-2020
https://doi.org/10.1109/SP40000.2020.00041
Cheng BTong QWang JTian W(2019)Malware Clustering Using Family Dependency GraphIEEE Access10.1109/ACCESS.2019.29140317(72267-72272)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2914031
Lu KPakki AWu Q(2019)Automatically Identifying Security Checks for Detecting Kernel Semantic BugsComputer Security – ESORICS 201910.1007/978-3-030-29962-0_1(3-25)Online publication date: 23-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-29962-0_1
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents