I wounder if I can solve this optimization problem through CVX ?

For a given values for t \in (1,...N) and P and the problem is

maximize \sum_{n=1}^{N} log det (X^{H}_{t}R_{n}X^{}_{t}+I_{t})

s.t. tr [X^{}_{t}X^{H}_{t}] \leq t P

Suppose we have different set of matrices R_{n}, where each R_{n} \in C^{N \times N} is a positive semidefinite, its diagonal elements are ones. Also each R_{n} is a rank deficient matrix.

I am expecting the output to be optimal matrix X_{opt}\in C^{N \times t}

How have you proven the objective function is concave and the constraints convex?

According to my calculations, the objective function is indefinite. Let X be 2 by 2, R = ones(2), which is psd with diagonal elements = 1, and is rank deficient. The Hessian of log(det(X'*R*X + eye(2))) evaluated at X = ones(2) has 2 zero eigenvalues, one positive eigenvalue, and one negative eigenvalue. Hence, is indefinite.

The problem you are discussing is non-convex. Methods for its exact or approximate solution are out of scope of this forum, as they do not involve CVX or TFOCS.