Learning by canonical smooth estimation. II. Learning and choice of model complexity

This paper analyzes the properties of a procedure for learning from examples. This “canonical learner” is based on a canonical error estimator developed in Part I. In learning problems one can observe data that consists of labeled sample points, and the goal is to find a model or “hypothesis” from a set of candidates that will accurately predict the labels of new sample points. The expected mismatch between a hypothesis prediction and the actual label of a new sample point is called the hypothesis “generalization error”. We compare the canonical learner with the traditional technique of finding hypotheses that minimize the relative frequency-based empirical error estimate. It is shown that for a broad class of learning problems, the set of cases for which such empirical error minimization works is a proper subset of the cases for which the canonical learner works. We derive bounds to show that the number of samples required by these two methods is comparable. We also address the issue of how to determine the appropriate complexity for the class of candidate hypotheses