Large Scale URL-based Classification Using Online Incremental Learning

We address the problem of large-scale topic classification of web pages based on the minimal text available in the URLs. This problem is challenging because of the sparsity of feature vectors that are derived from the URL text, and the typical asymmetry between the cardinality of train and test sets due to non-availability of sufficient sets of annotated URLs for training and very large test sets (e.g., in the case of large-scale focused crawling). We propose an online incremental learning algorithm which addresses these issues. Our experiments based on large publicly available datasets demonstrate an improvement of 0.11 -- 0.12 in terms of F-measure over the baseline algorithms, like Support Vector Machine, in difficult scenarios where the cardinality of train set is just a fraction of that of the test set.