How can i express this constraint using cvx?
Have you proven this is a convex constraint? It has terms of the form Something'*Something, where Something is linear in the optimization variable; and therefore is not linear (affine), at least as written.
i don’t know this constraint is convex or not. i don’t know how to do,
i only know \xi>0 and other variable is not related to $\mathbf{F}_{RF}$，this form is similar to the form of schur complement, but it can not transformed into LMI using schur complement.
As written, this is a Bilinear Matrix Inequality (BMI), which is non-convex. You might be able to do something with it using YALMIP.