Automated firmware testing using firmware-hardware interaction patterns

Firmware is low-level software which can directly access hardware and is often shipped with the hardware platform. This component of the system is increasing in scale and importance, and thus firmware validation is a critical part of system validation. Firmware validation relies on the interacting hardware components which are usually not available until the late design stages. This is generally addressed through co-simulating C/C++ based firmware code and HDL hardware models (including SystemC). However, this tends to be slow, and is further exacerbated by the large number of possible interleavings between the concurrent firmware and hardware threads. Typically, in the co-simulation, the scheduler, such as the SystemC scheduler, will only explore a single, or at best a small number of possible firmware-hardware interleavings and thus may miss critical bugs. In this paper we present an alternative approach to firmware validation that is based on automatically generating a test-set for the firmware with the goal of complete path coverage while considering its interactions with hardware and other firmware threads. It uses a service-function based Transaction Level Model (TLM) which has been used in the past for firmware-hardware codesign. The test generation is based on concolic testing which has been used successfully in software test generation. However, existing concolic testing tools are used for test-generation of sequential code, and cannot directly consider the interaction of other hardware/firmware threads with the target firmware thread during test generation. We address this limitation by exploiting specific interaction patterns between the firmware and hardware threads that can be analyzed from the TLM. We show how these patterns, along with the firmware and hardware threads are used to automatically generate a sequential program that is test-equivalent to the target firmware transaction and that can be used with a standard sequential program concolic test- generator. The tests generated can be (i) directly used for the firmware transaction and (ii) account for the multi-threaded interactions. These interaction patterns are practically relevant as they occur often in practice in real firmware benchmarks such as Linux device driver code, and its interacting QEMU emulated hardware code. Finally, we demonstrate the efficacy of our techniques for these benchmarks through a practical implementation that is automated and built on top of Frama-C, a static code analyzer, and KLEE, a concolic testing tool.