where, A&C&Dis a matrix of N\times N and X is matrix of N\times N and F(X)=trace$(AX^HX)$

how this problem is solved using cvx?

consider, constraint as : V(X)=trace$(CXDX^H)/(trace(CXX^H)+\sigma) >=\alpha$
this is a non convex quadratic constraint.
So,it can be changed to convex quadratic as

Y>=\alpha*(trace(CXX^H)+\sigma) and Y=trace(CXDX^H)
now the constraint become convex quadratic in (X,Y)

sir, I studied in a paper that $$\eqalignno{ &\sigma_{s}^{2}{\bf Y}(i, i)\geq\cr&\alpha_{i}\Bigg(\sigma_{s}^{2}\sum_{j\neq i}\langle { l}{i}{ l}{i}^{H}{ Xh}{j}{ h}{j}^{H}{ X}^{H}\rangle+\sigma_{{\rm re}}^{2}\langle { l}{i}{ l}{i}^{H}{ XX}^{H}\rangle+\sigma_{{\rm de}}^{2}\Bigg),\cr&\quad i=1,2, \ldots, { M}. }$$ is a convex quadratic in (X,Y). where Y(i,i) plays the role of trace(l_il_i^HXh_ih_i^HX^H).

Is this a wrong statement? And if it is a right statement, thn how this constraint is written in cvx?
the constraint can be written as:

if considering inequality as a convex by assuming anything, then how we can solve it. mainly I want to know to write objective function of such type (i.e X^HX)in cvx.

You are going about this entirely backwards. If you cannot prove this is convex, then you shouldn’t even be considering CVX. Prove to me that this model is convex first; then we will discuss how to express it in CVX. But as written, I am 99% certain it is not.

In that case, trace(CXDX^H) is indeed convex, but the constraint is not, because it is on the wrong side of the inequality. This remains a non-convex problem.

I see you’ve re-written your constraint; but I remain unconvinced that it is convex. How do you know it is convex? It is never a good sign that there are quadratic terms on the left-hand side of \geq.

The question isn’t whether it’s a convex function but a convex constraint. You have a convex quadratic on the left-hand side of \geq. That usually results in a non-convex constraints; and it always violates CVX’s rules.