How to express the X*X^H in CVX, where X is optimize variable, and X is an M*N matrix, X^H is the conjugate transpose of X

2

You don’t. See my answer at How to realize the trace(A*X*X^h*B^h) in CVX? - #2 by Mark_L_Stone . I.e., you can formulate trace(X*X') as square_pos(norm(X,‘fro’)), but you can't explicitly formulate X*X’` (for general purpose use).