Is the expression A X X ^ H A ^ H in the target function or in a constraint?
You may need to specify the related function or constraint, and then we can help you in a better way.
If you wan to take the trace of that, you can reformulate
Otherwise, it’s likely not convex, and you would need to use something other than CVX.
it appears that the argument of the norm is a matrix (even though there appears to be a typo in the dimensions you provided for
A and `X). Therefore, that is the operator two-norm, i.e., the largest singular value of the matrix argument. If that argument is not affine, which it isn’t here. I don’t believe that will be convex.