The simplified formulation looks like this:
PX
where P, X are matrix variables and X = xx’
one constraint I have is:
coefficients (PX, x) == Q
where Q is a linear matrix variable
The constraint is actually linear since I only care about the coefficient
but the formulation violates the DCP ruleset: (since X, P*X all nonlinear)
Only scalar quadratic forms can be specified in CVX
I wonder if there is a way to bypass this?

Besides, to deal with the coefficients, I find nonneg_poly_coeffs(n) and convex_poly_coeffs(n) might be helpful but does there exist a coefficient function for general polynomial? In my case, I have a matrix with each entry a general polynomial.

The problem can be solved using YALMIP but I get the wrong result, so I want to write code in CVX for debugging.

I have no idea what you claim the linear constraint is. Please show us a concrete example of what you claim is the linear constraint you want.

If you get the “wrong” result in YALMIP, perhaps you are either not specifying the problem correctly, or are obtaining a locally optimal solution which is not globally optimal, or the solver doesn’t even claim to have successfully solved the problem.

where \mu_{i,k} = x’Q_{i,k}x, Y, Q are matrix variables, x is vector variable. Z is known matrix.
The first term is a matrix with each entry a quadratic term in x with coefficients Y.
The second term is a matrix with each entry a quadratic term in x with coefficients (Q, Z).
The equality holds for all x, hence the coefficients of each entry should be zero.
I claim coeff(left part) == 0 a constraint linear in Y, Q.

x is either a variable or it is not. I still have no idea what your “theory” is. I don’t know what your mysterious coefficients operator is supposed to consist of - if you extract coefficients, there are no variables left (?) and therefore no meaningful constraint? Or you are somehow able to extract out one or more linear constraints which do with you want without use of any nonlinear equalities - if so, then go ahead and do so, and enter them into CVX.

A nonlinear matrix equality is not convex; you are more than welcome, however, to enter it as a constraint in YALMIP, isfused with a non-convex solver capable of handling quadratic equality constraints.

A trivial example:
let’s say all matrices are actually 1 by 1 (Y Q Z) and ignore [0 I 0] for now.
The equation above is then
2Yxx - \sum\sum xQxZ = 0
The coefficient operator with respect to x gives:
2Y - \sum\sum QZ = 0.
Since Z is known, this constraint is linear in Y, Z.
When (Y Q Z) is not 1 by 1, you get a matrix where each entry has a linear constraint similar to the above one.
Is this clear?

Maybe you need a symbolic (computer) algebra tool to extract the linear constraints you want. Then input those into CVX using appropriate numerically populated matrices and vectors to specify the linear constraints.

The bottom line: Yon can determine linear constraints however you please. Once you have done so, enter them in CVX in compliance with its syntax.