Classifying Natural Language Sentences for Policy

Organizations derive policies from a wide variety of sources, such business plans, laws, regulations, and contracts. However, an efficient process does not yet exist for quickly finding or automatically deriving policies from uncontrolled natural language sources. The goal of our research is to assure compliance with established policies by ensuring policies in existing natural language texts are discovered, appropriately represented, and implemented. We propose a tool-based process to parse natural language documents, learn which statements signify policy, and then generate appropriate policy representations. To evaluate the initial work on our process, we analyze four data use agreements for a particular project and classify sentences as to whether or not they pertain to policy, requirements, or neither. Our k-nearest neighbor classifier with a unique distance metric had a precision of 0.82 and a recall of 0.81, outperforming weighted random guess, which had a precision of 0.44 and a recall of 0.46. The initial results demonstrate the feasibility of classifying sentences for policy and we plan to continue this work to derive policy elements from the natural language text.