Sipping from the Firehose: How to "Grok" Gigabytes of Trace Debug Data

Long gone are the days when you could use a logic analyser on a CPU to find out what it is executing. Caches, out-of-order memory accesses and especially multi-processing all effectively hide the sequence of instructions that the CPU is working on. Traditional debug techniques with breakpoints and single-stepping help, but require interactive sessions at human rates rather than the gigahertz rates of today´s processors. To help this situation, many of today´s sophisticated processors come with a side-port (e.g. BDM, ETM, Nexus or Aurora) that the CPU can use to communicate what it is doing in real-time, at its full rate. By using a debug probe to capture this stream of information, the activities of the CPU can be traced. This trace data can be post-analysed to determine what the processor did, and even what memory values were processed. The problem is that at gigahertz processing speeds, that´s many gigabytes of data every second: what techniques are available to help understand this firehose of data? This trace data is one-dimensional: linear over time. So if a programming bug occurred during the trace capture, it is possible to see the sequence of steps leading up to it. Indeed, it is also possible to work backwards through the data looking for the exact condition that started the bug. The data is also suitable for use in profiling and coverage analysis: how often or whether certain routines were executed, and how long they took - all without instrumenting the code by adding time-wasting in-band profiling code, or by using statistical sampling techniques. Useful as this is, even more information can be gleaned by adding a second dimension: depth. More for optimising code than debugging it, being able to analyse call stacks can give insights into the (perhaps unexpected) way that the code base functions. However, to see code two-dimensionally requires a different view from the traditional source-based one: a graphical view, one that represents the call stack o- er time. This article explores these techniques, and explains how, through visualisation, the way a system is working (or not) can be more fully understood. Real-life examples of how using trace data helped debugging are also provided.