How to realize the trace(A*X*X^h*B^h) in CVX?

Nonconvex
2

If A = B, this can be reformulated as square_pos(norm(A*X,'fro')).

However, when A is different than B, it is not generally convex (or concave). Do you have a proof otherwise for your specific values of A and B ?

How to express the X*X^H in CVX, where X is optimize variable, and X is an M*N matrix, X^H is the conjugate transpose of X