Friendly and efficient message handling

Since communication software spends a significant amount of time on handling incoming messages, it is desirable that message handlers avoid expensive context-switches on frequently executed paths. High-performance active message systems demand that handlers run to completion without blocking. Unfortunately, disallowing all blocking in handlers makes it hard to integrate them into large, preemption-based systems, because each potentially blocking action, including library calls, must be rewritten. We have implemented a portable, hybrid upcall mechanism that is easier to use than active messages yet avoids unnecessary thread switching. The key idea is that message handlers are only allowed to block on locks protecting shared data. Inside message handlers, blocking on synchronous communication and condition variables is not allowed. This restriction allows most messages to be processed without unnecessary thread switching on the critical path. When a message handler has to suspend its work, it explicitly creates a continuation. We have added hybrid upcalls to the runtime system (RTS) of Orca, an object-based distributed shared memory system. By removing a thread switch from the critical path, remote object invocation latencies dropped by 300 μs. By building application-specific continuations rather than blocking a thread, we significantly reduced memory consumption by the Orca RTS. Finally, fewer undesired thread preemptions occur, because most messages are handled by a single thread