My objective function has a term of ‘Trace(ab)*Trace(ac)’, but it seems invalid and I tried the ‘quad_form’ command, it still doesn’t work. How to transfer it into a valid form.
You haven’t told us what the variables are. The first question is, is the expression convex or concave? if neither it can’t be used in CVX. If concave, it can’t be minimized in CVX. if convex, it can’'t be maximized in CVX.
cvx_begin
variables V(N+1,N+1) hermitian
dual variables t
maximize Pt(i)*real(trace(V*phi2))-r2*sigma*square(1+r2/(1+r2))*square(real(trace(V*phi1)))-y*(square(1+r2/(1+r2))*(r2*square(real(trace(V*R)))+real(trace(V*phi1))*real(trace(V*phi2))))
subject to
1- norm(V)>=0;
t: V <In> semidefinite(N+1);
cvx_end
The objective is concave.
Tr(ab)Tr(ac), the variable is a
That doesn’t look concave to me. Please show us your proof.
Also, your constraint norm(V)>=0 is not allowed in CVX, due to nominally being non-convex. However, it must hold even without being included; therefore it should be omitted.
The second derivative of Pt(i)*real(trace(V*phi2))-r2*sigma*square(1+r2/(1+r2))*square(real(trace(V*phi1))) to V is 0. and the second derivative of -y*(square(1+r2/(1+r2))*(r2*square(real(trace(V*R)))+real(trace(V*phi1))*real(trace(V*phi2)))) is <= 0. so it is concave. And my first constraint is 1- norm(V)>=0 not norm(V)>=0.
My apology for misreading that constraint.
As for the objective being concave, unless phi1 =- 0 (matrix?), I don’t see how the 2nd derivative (whatever exactly that means) of that first term is 0. In any event, you need to show that the Hessian with respect to V is negative semidefinite for all values of V.
oh, I’m sorry the 2nd derivative of that first term is -2*(1+r2/(1+r2))^2phi1phi1 is negative and the 2nd derivative of the second term is -y2(1+r2/(1+r2))^2phi1’phi2’. So the Hessian is negative semidefinite. And I have tried that if I eliminate one of trace(Vphi1) or trace(Vphi2), the problem can be sovled by CVX 
And I think real(trace(V*phi1))real(trace(Vphi2)) can be treated as a scalar, the problem is the form. Thanks for your patient!
Joint concavity with respect to all the matrix elements is needed. The Hessian being negative semidefinite does that; The Hessian is not s scalar. And that is needed, despite the expression evaluating to a scalar.
You also have not yet shown us what ph1i and phi2 are.
phi1 and phi2 are two PSD matrixs. I thought real(trace(Vphi1))real(trace(Vphi2)) can be treated as another quadratic form, similarly like square(real(trace(Vphi1))) 
No, it can not. There is a reason CVX does not allow matrix quadratics. They are very different from scalar quadratics.
trace(V*phi1) is an affine function of V, which when squared, is a convex quadratic. Of course, that is not concave.
trace(V*phi)*trace(V*phi2) when ph1 \ne phi2 is generally neither convex nor concave.
Thank you! So, you mean it won’t work, but why is the Hessain of trace(V*phi)trace(Vphi2) is PSD if it is generally neither convex nor concave? I’m just starting convex optimization problems not so long.
Please show me why the Hessian is PSD. It is not.
The Hessian an N+1 by N+1 matrix. You may wish to review https://en.wikipedia.org/wiki/Hessian_matrix .
I mean because the 2nd derivative of the second term is -y 2 (1+r2/(1+r2))^2 *phi1’ *phi2’. where phi1 and phi2 are two PSD and y and r2 are both positive constant. So the second term is concave.
That is wrong. if you need further understanding of that, perhaps review multivariable calculus and seek help at https://math.stackexchange.com/ …
At this point, I think your CVX question has been resolved.
Ok, Thank you very much again!