Inducing models less greedily

Most algorithms which induce model structure from sample data proceed “greedily” to varying degrees. That is, they sequentially add to the current model the candidate component which works best with the existing structure. This greedy search procedure is relatively fast, but is not optimal, as there can exist models within the “reachable” space which have less complexity and/or greater accuracy on the training data. Indeed, this difference in training performance between optimal and greedy models can be large. We review example effects of greediness in regression to motivate study of the issue with another popular model form: decision trees. A new tree algorithm, “Texas Two-Step”, is introduced which looks ahead one more generation than standard procedures. In other words, it judges a potential split not by how the resulting child nodes turn out, but by how the grandchildren do. Preliminary results are compared on a recent field application: identifying a bat´s species by its chirps