How do I express X=x*x‘

You can’t explicitly form X=x*x' in CVX unless x is a row vector; I presume your x is a column vector, which would result in x*x' being non-convex. Depending on how x*x' is going to be used, it’s possible there is some formulation in CVX which does not involve directly forming x*x'. For example, trace(x*x') can be formulated as square_pos(norm(x,'fro')).

Your fist task is to prove that the optimization problem you wish to enter in CVX is convex. Otherwise, CVX is not appropriate for the problem.