An indefinite stochastic linear quadratic optimal control problem for the FBSDE system with jumps

This paper is concerned with an indefinite linear quadratic (LQ) optimal control problem for the forward-backward stochastic system with jumps. Based on positive definite LQ problems, we give a more general condition, under which the control weight cost need not to be positive. In this paper, a relax compensator plays an important role in solving the indefinite LQ problems. Moreover, we discuss the corresponding stochastic Hamiltonian system and give a representation of the open-loop control. The main results in this paper can also be used to solve more general problems in stochastic control theory.