Post-hoc analyses in multiple regression based on prediction error

A well-known problem in multiple regression is that it is possible to reject the hypothesis that all slope parameters are equal to zero, yet when applying the usual Student’s T -test to the individual parameters, no significant differences are found. An alternative strategy is to estimate prediction error via the 0.632 bootstrap method for all models of interest and declare the parameters associated with the model that yields the smallest prediction error to differ from zero. The main results in this paper are that this latter strategy can have practical value versus Student’s T ; replacing squared error with absolute error can be beneficial in some situations and replacing least squares with an extension of the Theil–Sen estimator can substantially increase the probability of identifying the correct model under circumstances that are described.