The limiting spectral distribution of the product of the Wigner matrix and a nonnegative definite matrix

Let W n be n × n Hermitian whose entries on and above the diagonal are independent complex random variables satisfying the Lindeberg type condition. Let T n be n × n nonnegative definitive and be independent of W n . Assume that almost surely, as n → ∞ , the empirical distribution of the eigenvalues of T n converges weakly to a non-random probability distribution. n = n − 1 / 2 T n 1 / 2 W n T n 1 / 2 . Then with the aid of the Stieltjes transforms, we show that almost surely, as n → ∞ , the empirical distribution of the eigenvalues of A n also converges weakly to a non-random probability distribution, a system of two equations determining the Stieltjes transform of the limiting distribution. Important analytic properties of this limiting spectral distribution are then derived by means of those equations. It is shown that the limiting spectral distribution is continuously differentiable everywhere on the real line except only at the origin and that a necessary and sufficient condition is available for determining its support. At the end, the density function of the limiting spectral distribution is calculated for two important cases of T n , when T n is a sample covariance matrix and when T n is the inverse of a sample covariance matrix.