A proposed metric for the evaluation of natural language systems

It is proposed that a simple number scale be established for the evaluation of natural-language text-processing systems. This metric is to be based on human linguistic performance, taken as 1.0, and is the simple average of four subcomponents, i.e. the size of the lexicon, the speed and accuracy of the parse, and the overall experience of the system. The metric would represent the ability to parse English text at the rate of 12 sentences per minute with perfect accuracy based on a vocabulary of 25000 words and an experience base of 40 million sentences. It is noted that, if such a metric can be successfully implemented for natural-language projects it can then serve as a model for the development of validation techniques for other hard-to-evaluate stress in artificial intelligence such as expert system, learning, or pattern-recognition programs