this is the square root of the quadratic form you’ve asked for, but the optimization problem is equivalent—and it is always good to avoid quadratic forms when possible; see this section of the users guide.

Unfortunately, you have a much bigger problem: your constraint is not convex. For one thing, requiring that the expression X’HX-R be positive or negative semidefinite is non-convex. If you are concerned only with the PSD case, I believe you can use Schur complements to get a convex result; for example:

[ R, X' ; X, inv(H) ] == semidefinite(m+n)

You’ll need to double check this yourself. But I actually don’t think you can handle the NSD case.

chuzheng7, there is a problem with your code: R is not PSD. But if the original question is not convex, he’s stuck. CVX is strictly for convex problems.