Micro-architectural anatomy of a commercial TCP/IP stack

Over the last couple of decades, computer architects and performance analysts have routinely attempted to profile the overhead of TCP/IP processing in an effort to understand where the time was spent. It is well understood that this is a rather difficult problem since the processing time is spread across various software modules such as the network stack, interrupt routines, drivers, O/S scheduler, etc. As a result, the problem of extracting the micro-architectural characteristics of TCP/IP processing is significantly more challenging. In this paper, we start by covering the previous attempts at this problem and show what existing tools can provide in terms of execution time characteristics. We then propose a detailed methodology that combines full-system simulation, cycle-accurate performance simulations and symbol annotation to provide a rich cycle-accurate view of TCP/IP packet processing execution. We discuss initial results based on our profiling methodology and discuss where the time is spent. This includes an analysis of micro-architectural characteristics (such as instruction breakdown, CPI, MPI and TLB misses on a state-of-the-art microprocessor).