Optimal stopping and effective machine complexity in learning

We study learning in a general class of machines which return a (variable) linear form of a (fixed) set of nonlinear transformations of points in an input space. A fixed machine in this class accepts inputs X from an arbitrary input space and produces scalar outputs Y=Σ_i=1^dψ_i(X)w_i *+ξ=ψ(X)´w*+ξ (1). Here, w*=(w₁*,…,w _d*)´ is a fixed vector of real weights representing the target concept to be learned, for each i, ψ_i(X) is a fixed real function of the inputs, with ψ(X)=(ψ₁(X),…ψ_d(X))´ the corresponding vector of functions, and ξ is a random noise term. We suppose that the learner receives an i.i.d., random sample of examples (X₁,Y₁),…,(X_n,Y_n) generated according to the joint distribution on input-output pairs (X,Y) induced through the medium of the (unknown) relation (1) and a fixed (unknown) distribution on input-noise pairs (X,ξ). The goal of the learner is to infer a hypothesis w=(w₁,…w_d )´ with small (mean-square) generalisation error E(Y-ψ(X)´w) ² on future random examples (X,Y) generated independently of the training sample from the same underlying distribution. Here E denotes expectation with respect to the underlying probability distribution generating the examples