A general maximum principle for optimal control of forward–backward stochastic systems

A general maximum principle for optimal control problems derived by forward–backward stochastic systems is established, where control domains are non-convex and forward diffusion coefficients explicitly depend on control variables. These optimal control problems have broad applications in mathematical finance and economics such as the recursive mean–variance portfolio choice problems. The maximum principle is applied to study a forward–backward linear-quadratic optimal control problem with a non-convex control domain; an optimal solution is obtained.