Network security via reverse engineering of TCP code: vulnerability analysis and proposed solutions

The Transmission Control Protocol/Internet Protocol (TCP/IP) suite is widely employed to interconnect computing facilities in today´s network environments. However, there exist several security vulnerabilities in the TCP specification and additional weaknesses in a number of its implementations. These vulnerabilities may allow an intruder to “attack” TCP-based systems, enabling him/her to “hijack” a TCP connection or cause denial of service to legitimate users. The authors analyze the TCP code via a “reverse engineering” technique called “program slicing” to identify several of these vulnerabilities, especially those that are related to the TCP state-transition diagram. They discuss many of the flaws present in the TCP implementation of many widely used operating systems, such as SUNOS 4.1.3, SVR4, and ULTRIX 4.3. The corresponding TCP attack “signatures” (including the well-known 1994 Christmas Day Mitnick Attack) are described, and recommendations are provided to improve the security state of a TCP-based system (e.g., incorporation of a “timer escape route” from every TCP state). Also, it is anticipated that wide dissemination of this article´s results may not only lead to vendor patches to TCP code to plug security holes, but also raise awareness of how program slicing may be used to analyze other networking software and how future designs of TCP and other software can be improved