The off-diagonal elements of this construction are non-convex. Of course, since it’s not a scalar, you can’t really be using this un-modified in your objective anyway. What exactly are your trying to do? It may be convex, but there is no way to know without more info.

You know, in CS model, y=Acs*x+n, I try to calculate the covariance matrix(RR) of y to represent the mutual information between y and x. And in that case,what can I do to solve this problem, Can I code it in another form?

As mcg has said, you haven’t told us what the objective function is. Are you trying to optimize with respect to the matrix phi, some scalar function of RR, which as you have written is just the formula for covariance of a linear transformation of a random variable having covariance Rx?

Yes , I am trying to optimize with respect to the matrix phi. the objective function is the maximum of mutual information. And the MI(mutual information)=log[det(RR+Rn)]

Hmm. I’m not sure if this is convex or not. If it is, it will need to be expressed using linear matrix inequalities, likely using some sort of schur complement approach. I’m really not sure.