I had to note that the R is hermitian too, so the trace(RV) is real. But R is not a variable which needs to be optimize. I don’t know how to mark R as a hermitian matrix.

g = rand(N,1,M,K)+1j*rand(N,1,M,K); # g represents M*K vectors with N dimensions
gt = pagectranspose(g);
R0 = pagemtimes(g,gt); # R0 represents M*K hermitian matrixs
# we take R0 as a const parameter in CVX
cvx_begin
...
for i,j
TR(i,j) = trace(R0(:,:,i,j)*V(:,:,i,j));
for i
R(i)=sum(TR(i,:));
subject to
sum(-rel_entr(1, ones+R)) - sum(a(:,i).*(R-TR(:,i))+b(:,i)) >= yita;
# where ones denotes a vector of ones, and 'a' 'b' denote const real parameters too.
...
end_cvx