A Comprehensive Solution for Deterministic Replay Debugging of SoftPLC Applications

Deterministic replay debugging is an approach to finding bugs in deployed software. It records an application run in the field so that it can deterministically be replayed offline in a development system for debugging purposes. To enable deterministic replay debugging, it is necessary to record all external influences and sources of nondeterminism in the original program run. From that trace log and from a known initial state, the program can be replayed deterministically without requiring any connection to the original environment. In this paper, we present a solution for deterministic replay debugging of hard real-time multitasking SoftPLC applications written in the IEC 61131-3 languages. By taking advantage of the special properties of these programs and by careful engineering, our technique allows recording a SoftPLC application run in the field with minimal overhead and obeying real-time constraints. In later phases, which are offline, the original program run is reconstructed from the minimal information recorded so it can be replayed for debugging. In comparison to previous work, our solution has several advantages: Instead of recording task scheduling information, it reconstructs the task interleaving based on data dependencies, thereby significantly simplifying the recording phase. Additionally, it incorporates a technique for periodically capturing the complete internal state of the system, which can later be used as a starting point for replay. We present the conceptual basis of our approach, a tool chain which provides deterministic replay debugging to the user as a set of fully automated tools, and an evaluation as well as an industrial case study for validating the approach.