How to realize the multiplication of a optimization variable in CVX? Or is there another way(not restricted to CVX) to realize it?

Is the expression A X X ^ H A ^ H in the target function or in a constraint?
You may need to specify the related function or constraint, and then we can help you in a better way.

If you wan to take the trace of that, you can reformulate trace(A*X*(A*X')) as square_pos(norm(A*X,'fro'))

Otherwise, it’s likely not convex, and you would need to use something other than CVX.


it appears that the argument of the norm is a matrix (even though there appears to be a typo in the dimensions you provided for A and `X). Therefore, that is the operator two-norm, i.e., the largest singular value of the matrix argument. If that argument is not affine, which it isn’t here. I don’t believe that will be convex.