Language identification: A new fast algorithm to identify the language of a text in a multilingual corpus

Identifying the language of a text is a very important preliminary phase in the categorization of multilingual documents or even in information retrieval. This phase becomes difficult if we just consider the word as a basic unit of information in texts. Because It could be possible for some languages as French or English but very difficult for some other languages as German, Chinese and Arabic. In this paper, we present the most known identification algorithms, and we propose a new fast and effective algorithm based on n-grams of characters. We also evaluate the obtained results with other algorithms when using the two approaches of texts segmentation: words approach, n-grams approach.