Normally, x*y is non-convex when both x and y are continuous and it makes problem intractable. However, it really appears frequently in application. If we have more constraints, like the range of one variable, i.e., 0<x<1, can we transform it into convex form? or decouple these two variables?

It appears your question was not completed. Please do so.

Unless x*y is part of some larger expression which overall is convex or concave, then it is indefinite, and therefore neither convex nor concave.

The assumption of convexity is absolutely fundamental to CVX’s design. That’s not going to change.

But there are cases where products of different variables do readily lead to convex models. For instance, consider $$\prod_{i=1}^n x_i \geq \alpha$$ with x nonnegative and \alpha\geq 0. This can be converted to an equivalent convex form $$\left(\prod_{i=1}^n x_i \right)^{1/n} \geq \alpha^{1/n}$$ The left-hand quantity is a geometric mean, which is concave and supported by CVX.

Thanks for your reply. What you mentioned is really meaningful. Let me keep it in my tool box.