x'*A*y is an indefinite form, being neither convex nor concave, and this is not allowed in CVX. For instance, if A = 1 and x and y are scalars, this comes out to x*y , Additionally, you wold not be allowed in CVX to use norm on it given it is not affine.
Please read Why isn't CVX accepting my model? READ THIS FIRST! .