Penalized learning as multiple object optimization

Learning algorithms guided by costs with a variety of penalties are discussed. Both unsupervised and supervised cases are addressed. The penalties are added and/or multiplied to the basic error measure. Since these extra penalties include combination parameters with respect to the basic error, the total problem belongs to a class of multiple object optimization. Learning algorithms in general cases are derived first. Then, individual cases such as penalties on undesirable weights and outputs are treated. A method to find a preferred solution among the Pareto optimal set of the multiple object optimization is given