Quadratic constraint of a matrix is not allowed in cvx?

Your constraints are not convex.

Have you seen the SDP formulation, as a linear (convex) SDP? Perhaps there is an SDP formulation which is a convex relaxation of the original problem?

It appears that you haven’t read the CVX FAQ Why isn’t CVX accepting my model? READ THIS FIRST! .