Fine-grain software distributed shared memory on SMP clusters

Commercial SMP nodes are an attractive building block for software distributed shared memory systems. The advantages of using SMP nodes include fast communication among processors within the same node and potential gains from clustering where remote data fetched by one processor is used by other processors on the same node. This paper describes a major extension to the Shasta distributed shared memory system to run efficiently on a cluster of SMP nodes. The Shasta system keeps shared data coherent across nodes at a fine granularity by inserting inline code that checks the cache state of shared data before each load or store in an application. However allowing processors to share memory within the same SMP is complicated by race conditions that arise because the inline state check is non-atomic with respect to the actual load or store of shared data. We present a novel protocol that avoids such race conditions without the use of costly synchronization in the inline checking code. To characterize the benefits of using SMP nodes in the context of Shasta, we present detailed performance results for nine SPLASH-2 applications running on a cluster of Alpha multiprocessors