The off-diagonal elements of this construction are non-convex. Of course, since it’s not a scalar, you can’t really be using this un-modified in your objective anyway. What exactly are your trying to do? It may be convex, but there is no way to know without more info.
You know, in CS model, y=Acs*x+n, I try to calculate the covariance matrix(RR) of y to represent the mutual information between y and x. And in that case,what can I do to solve this problem, Can I code it in another form?
As mcg has said, you haven’t told us what the objective function is. Are you trying to optimize with respect to the matrix phi, some scalar function of RR, which as you have written is just the formula for covariance of a linear transformation of a random variable having covariance Rx?