A Formal Methodology for Network Protocol Fingerprinting

Network protocol fingerprinting refers to the process of identifying a protocol implementation by their input and output behaviors. It has been regarded as both a potential threat to network security and also as a useful mechanism for network management. Existing protocol fingerprinting tools share common disadvantages such as being protocol-specific and difficult to automate. This paper proposes a formal methodology for fingerprinting experiments using which we can model a broad spectrum of fingerprinting problems and design-efficient algorithms. We present a formal behavioral model that specifies a protocol principal by its states and transitions, then identify a complete taxonomy of fingerprint matching and discovery problems is identified based on 1) whether the fingerprinting experiment is active or passive and 2) the information available about the specifications and implementations. Algorithms to solve the problems are discussed. In particular, for fingerprint matching algorithm, we propose an efficient PEFSM online separation algorithm for active experiment and concurrent passive testing for passive experiments. For fingerprint discovery problem, there are two cases: if the protocol specification is available as a nondeterministic PEFSM, we apply across verification and back-tracing technique for active and passive discovery, respectively; if no specification is available, we take the machine learning approach and discover the fingerprint by active testing.