Regularized gap function as penalty term for constrained minimization problems

By using the regularized gap function for variational inequalities, Li and Peng introduced a new penalty function P α ( x ) for the problem of minimizing a twice continuously differentiable function in closed convex subset of the n-dimensional space R n . Under certain assumptions, they proved that the original constrained minimization problem is equivalent to unconstrained minimization of P α ( x ) . The main purpose of this paper is to give an in-depth study of those properties of the objective function that can be extended from the feasible set to the whole R n by P α ( x ) . For example, it is proved that the objective function has bounded level sets (or is strongly coercive) on the feasible set if and only if P α ( x ) has bounded level sets (or is strongly coercive) on R n . However, the convexity of the objective function does not imply the convexity of P α ( x ) when the objective function is not quadratic, no matter how small α is. Instead, the convexity of the objective function on the feasible set only implies the invexity of P α ( x ) on R n . Moreover, a characterization for the invexity of P α ( x ) is also given.