Using the Levenshtein Edit Distance for Automatic Lemmatization: A Case Study for Modern Greek and English

In the present work we have implemented the Edit Distance (also known as Levenshtein Distance) on a dictionary-based algorithm in order to achieve the automatic induction of the normalized form (lemma) of regular and mildly irregular words with no direct supervision. The algorithm combines two alignment models based on the string similarity and the most frequent inflexional suffixes. In our experiments, we have also examined the language-independency (i.e. independency of the specific grammar and inflexional rules of the language) of the presented algorithm by evaluating its performance on the Modern Greek and English languages. The results were very promising as we achieved more than 95 % of accuracy for the Greek language and more than 96 % for the English language. This algorithm may be useful to various text mining and linguistic applications such as spell-checkers, electronic dictionaries, morphological analyzers, search engines etc.