x'*Q*x is non-convex (and non-concave) if both x and Q are optimization variables or expressions involving them.
CVX is a tool for convex optimization, not bi-convex optimization.
if you’re willing to use Python, you could consider CVXPY with its DMCP multi-convex optimization add-on GitHub - cvxgrp/dmcp: A CVXPY extension for multi-convex programming . Also see https://web.stanford.edu/~boyd/papers/pdf/dmcp.pdf In Julia, there is a MultiConvex add-on for Convex.jl. Or you could hatch your own iterative scheme in CVX, in which case you’d be on your own in getting it robust enough to work reliably to even find a local optimum.
Or just use a non-convex solver, for instance under YALMIP.