Analysis of Header Usage Patterns of HTTP Request Messages

The headers used by the various network protocols represent a good source of information for understanding the behavior and the properties of the traffic and detecting potential security attacks. In this paper we present a comprehensive analysis of the usage patterns of the headers included in the HTTP request messages of the clients. Our study shows that message headers vary significantly in terms of number, field names and field values. In general, Web browsers tend to specify in detail the client preferences by including in the request messages a large and variable number of headers. The usage patterns of these headers together with their field values provide some useful hints for website customization. On the contrary, the requests issued by Web robots are characterized by some distinctive patterns specific of the individual robots. These one-to-one correspondences between patterns and Web robots can then be used for their identification.