Sparse Logistic Discriminant Analysis

Linear discriminant analysis (LDA) is a well-known method to extract efficient features for multi-class classification. Otsu derived the optimal (ultimate) non-linear discriminant analysis (ONDA) by supposing underlying probabilities and showed that ONDA was closely related to Bayesian decision theory (posterior probabilities). Also Otsu pointed out that the usual LDA could be regarded as the linear approximation of this ultimate ONDA through the linear approximations of the Bayesian posterior probabilities. This theory of ONDA suggests that we can construct a novel nonlinear discriminant mapping by utilizing the estimates of the posterior probabilities. Based on this theory, logistic discriminant analysis (LgDA) was proposed by one of the authors as the approximation of ONDA. In LgDA, the posterior probabilities are estimated by logistic regression. In this paper, we propose the sparse logistic discriminant analysis in which the posterior probabilities are estimated by the sparse logistic regression with L2-or L1-regularizer to improve the generalization performance of LgDA further. Experiments using the standard datasets for classification reveal that the discriminant spaces by our proposed method (LgDA-L2 and LgDA-L1) are better than those by LDA and LgDA in terms of the recognition rates for test samples.