Enhanced Privilege Separation for Commodity Software on Virtualized Platform

Conventional privilege separation can effectively reduce the TCB size by granting privilege to only the privileged compartments. However, since they this approach relies on process isolation to ensure security assurance, malware exploiting against kernel components can easily compromise. Meanwhile, the frequent inter-process communications between separated processes inevitably incur notable overhead. To ameliorate these problems, we propose to perform privilege separation without partitioning application into two processes. Instead, we leverage virtualization to enforce the isolation of sensitive portions from other untrusted code. The virtual machine monitor intercepts all the code context switches transparently without requiring the application to explicitly use IPC as privilege context transition. We have implemented a prototype of our system, named Coir, based on commodity hypervisor Xen. Evaluation of our prototype includes a real-world remote control application, which is partitioned and protected in Coir-enabled hypervisor on unmodified Windows XP. We discuss the isolation strength as well as the performance penalty of our system based on the practical case.