this constraints cannot be put in cvx with the form w^2<=xy;
it says ??? Error using ==> cvx.times at 262
Disciplined convex programming error:
Invalid quadratic form(s): not a square.
how can I modify the form?
Indeed, this constraint does not obey the disciplined convex programming rules; CVX is right to reject it. In particular, the right-hand side is not concave. But you might try this:
norm([Pij(ii),Qij(ii)]) <= geo_mean([lij(ii),V(mpc.branch(ii,1)])
Note that this forces lij(ii) and V(mpc.branch(ii,1)) to be nonnegative. If that is what you intend to be true, this is the correct reformulation. If not, then your problem is almost certainly not convex.