however in cvx, I tried to use norm(h’ * v * v’), but it gives an error. It seems that CVX only accepts a square matrix. Is there any syntax I can use? at least any syntax i can use to make it work? approximation is okay as well. Due to the coupling of variables, I can not transform the vector v in to matrix V=v*v^H (like SDR).
That error message basically says you have one or more quadratic elements which are not recognizably convex due to being squares. It has nothing to do with whether a matrix has square dimensions.
You haven’t followed DCP rules. Moreover, your expression is non-convex (unless h is all zeros, or N=1), so reformulation to be acceptable to CVX is not possible. If there is a convex approximation suitable for your purposes, that is for you to determine, because it’s your problem, so you presumably know why you’re trying to solve it.