Paging tradeoffs in distributed-shared-memory multiprocessors

Massively parallel processors have begun using commodity operating systems that support demand paged virtual memory. To evaluate the utility of virtual memory, we measured the behavior of seven shared memory parallel application programs on a simulated distributed shared memory machine. Our results: confirm the importance of gang CPU scheduling; show that a page faulting processor should spin rather than invoice a parallel context switch; show that our parallel programs frequently touch most of their data; and indicate that memory, not just CPUs, must be “gang scheduled”. Overall, our experiments demonstrate that demand paging has limited value on current parallel machines because of the applications´ synchronization and memory reference patterns and the machines´ high page fault and parallel context switch overheads