Mismatched hypothesis testing with application to digital modulation classification

This paper considers the problem of mismatched hypothesis testing, where approximate likelihood functions are used instead of true likelihood functions. Given a hypothesis testing problem, the maximum likelihood (ML) solution is known to be optimal when true likelihood functions are used, but the optimality does not hold anymore if mismatched approximate likelihood functions are employed instead, in order to reduce computational complexity, for instance. In this paper, we investigate the mismatched ML framework using approximate likelihood functions, while the mismatches between the true and the approximate likelihood functions are corrected by additive compensating constants. The probability of error of this mismatched hypothesis testing is analyzed asymptotically, assuming a large number of samples, and the compensating constants that maximize the error exponent are established. The general results on the mismatched hypothesis testing are then utilized in designing and optimizing a digital modulation classifier with low complexity.