Encounter error "Invalid quadratic form(s): not a square." when trying to multiply variables

You need to express that as a quadratic forum. If you do that, or just calculate the Hessian of x1^2+y1^2-4*x*y, you will see that the Hessian is indefinite, so that the expression is neither convex nor concave. Therefore it can’t be used in CVX.