Is the expression A X X ^ H A ^ H in the target function or in a constraint?
You may need to specify the related function or constraint, and then we can help you in a better way.
If you wan to take the trace of that, you can reformulate trace(A*X*(A*X')) as square_pos(norm(A*X,'fro'))
Otherwise, it’s likely not convex, and you would need to use something other than CVX.
it appears that the argument of the norm is a matrix (even though there appears to be a typo in the dimensions you provided for A and `X). Therefore, that is the operator two-norm, i.e., the largest singular value of the matrix argument. If that argument is not affine, which it isn’t here. I don’t believe that will be convex.