Is the expression A X X ^ H A ^ H in the target function or in a constraint?

You may need to specify the related function or constraint, and then we can help you in a better way.

If you wan to take the trace of that, you can reformulate `trace(A*X*(A*X'))`

as `square_pos(norm(A*X,'fro'))`

Otherwise, it’s likely not convex, and you would need to use something other than CVX.

it appears that the argument of the norm is a matrix (even though there appears to be a typo in the dimensions you provided for `A`

and `X). Therefore, that is the operator two-norm, i.e., the largest singular value of the matrix argument. If that argument is not affine, which it isn’t here. I don’t believe that will be convex.