Exploiting language cluster information for language pair identification

Recently, significant progress has been witnessed in automatic language identification. Perhaps the most important development is the application of discriminative training. However, efficient and effective discriminative training for language pair identification is still a challenging problem. Specifically, with the increase of the number of languages to be recognized, the number of the language pairs will grow quadratically, which may make the discriminative training complex. Furthermore, it is difficult to collect data for certain languages. The data imbalance also biases the discriminative training and degrades the performance. To address these issues, we propose to exploit language clustering information for effective language pair identification. It is mainly motivated from the perspective of the linguistics that all languages can be divided into several families. In our proposed method, the language clusters are constructed based on the linguistic knowledge. The language pairs are categorized into “intra-cluster” and “inter-cluster”, and different weighting schemes are proposed for identification. To evaluate the effectiveness of our proposed scheme, we conduct extensive experiment on NIST LRE 2011 and a small subset of languages selected from it. The experimental results show that by exploiting the language clustering information, the resulted identification performance has been improved significantly, compared with baseline system.