That error message basically says you have one or more quadratic elements which are not recognizably convex due to being squares. It has nothing to do with whether a matrix has square dimensions.
You haven’t followed DCP rules. Moreover, your expression is non-convex (unless h is all zeros, or N=1), so reformulation to be acceptable to CVX is not possible. If there is a convex approximation suitable for your purposes, that is for you to determine, because it’s your problem, so you presumably know why you’re trying to solve it.