Protocol State Machines and Session Languages: Specification, implementation, and Security Flaws

Input languages, which describe the set of valid inputs an application has to handle, play a central role in language-theoretic security, in recognition of the fact that overly complex, sloppily specified, or incorrectly implemented input languages are the root cause of many security vulnerabilities. Often an input language not only involves a language of individual messages, but also some protocol with a notion of a session, i.e. A sequence of messages that makes up a dialogue between two parties. This paper takes a closer look at languages for such sessions, when it comes to specification, implementation, and testing - and as a source of insecurity. We show that these ´session´ languages are often poorly specified and that errors in implementing them can cause security problems. As a way to improve this situation, we discuss the possibility to automatically infer formal specifications of such languages, in the form of protocol state machines, from implementations by black box testing.