Base paper in which the following optimization question is given

objective function::: $$\mathop{\min}{{ X,Y}}{ P}{{\rm tot}}({ X})+\mu(\langle\bar{{ Y}}\rangle-\lambda_{\max} (\bar{{ Y}})) : $$
Constraint:: $$\eqalignno{ &\bar{{ Y}}:=\left[\matrix{{ Y} & {\cal L}({ X})\cr{\cal L}^{H}({ X}) & 1}\right] \succeq 0}$$
and

$$\eqalignno{ &\sigma_{s}^{2}{\bf Y}(i, i)\geq\cr&\alpha_{i}\Bigg(\sigma_{s}^{2}\sum_{j\neq i}\langle { l}{i}{ l}{i}^{H}{ Xh}{j}{ h}{j}^{H}{ X}^{H}\rangle+\sigma_{{\rm re}}^{2}\langle { l}{i}{ l}{i}^{H}{ XX}^{H}\rangle+\sigma_{{\rm de}}^{2}\Bigg),\cr&\quad i=1,2, \ldots, { M}.}$$

The above defined are the function to calculate optimized value:
$${ P}{{\rm tot}}({ X})=\langle(\sigma{s}^{2}{ HH}^{H}+\sigma_{{\rm re}}^{2}I_{N}){ X}^{H}{X}\rangle.$$ which is a convex quadratic function and $${\cal L}({ X})=({ l}{1}^{H}{ Xh}{1}, { l}{2}^{H}{ Xh}{2}, \ldots, { l}{M}^{H}{ Xh}{M})^{T}$$ where l_1,l_2…l_M and h_1,h_2,…h_M is a N * N matrix. H is a N * M matrix and Y is a M * M matrix. and all other sigma’s is a constant value. and alpha is a threshold.
The above described objective function is a difference of two conex fuction.

Thank you sir for your reply.Yes sir I have the original paper with me. “http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6463493” . File is not getting uploaded. Sir, the paper name is “Iterative D.C. Optimization of Precoding in
Wireless MIMO Relaying” , an ieee journal.

Sir I have written a code for this except of that second constraint because I am not getting that how should I code X * X^H. But thn also it is not working. I am attaching the code. It gives the error in quad_form.

M=3;
N=5;
R=1;

n_r=N/R;   

%since N=R*n_r

sig=10^(20/10)   ;  %variance of signal

s_r=10^(0/10)   ; %variance of received signal

iden=eye(N);    %identity function of n*n matrix

sigde=10^(0/10);

SINR_db=[1:10];

%%

for i=1:M

    l_temp=conj(transpose((randn(1,N) + sqrt(-1) * randn(1,N)) / sqrt(2)));
    k(:,:,i)=l_temp;
    down(:,i)=k(:,:,i);
    
    h_temp=transpose((randn(1,N) + sqrt(-1) * randn(1,N)) / sqrt(2));
    m(:,:,i)=h_temp;
    upli(:,i)=m(:,:,i);

end

P=(sig*upli*conj(transpose(upli))+s_r*iden(N));

for i=1:M

    lin=conj(transpose(down(:,i)))*upli(:,i);
    li(i)=lin;
   
end

L=transpose(li);

Y=L*conj(transpose(L));

P=(sig*upli*conj(transpose(upli))+s_r*iden(N));

cvx_begin 

   variable X(N,N) 

   variable Y(M,M)

    for i=1:M

    li=conj(transpose(down(:,i)))*X*upli(:,i);
    lit(i)=li;

   end

L=transpose(lit);

%Y=L*conj(transpose(L));

 Ycap=[ Y, L ; conj(transpose(L)), eye(1) ];

   minimize(trace(real(quad_form(X,P))+0.5*(trace(Ycap)-lambda_max(Ycap))))

   subject to

   [ Y, L ; conj(transpose(L)), eye(1) ] == hermitian_semidefinite(M+1)

   norm(X,Inf) <= 1;

   norm(Y,INF)<=1;

  %X>=0

  cvx end

Sir, I need a favour from you… I have send the paper "Iterative D.c optimization of Precoding in Wireless MIMO Relaying " Please will you tell me that the objective function and constraint of equation 15, 22, 24, 25, 28 and 34 are either convex function or non convex. And also refer me some source from where I can easily differentiate between convex function and non convex function. Sir, its my kind request to you…

It is not at all clear you can solve this with CVX. Have you proven that it is convex? More specifically, have you proven that it can be expressed as a semidefinite program? To be fair, it is very difficult to read, so I might be missing something, but presence of nonconvex quadratics is a problem.

Do you mean to take the trace of the quantities inside angle brackets? You should use the trace identity to commute some of those products to convert them to scalar quadratic forms. If you have an original paper to refer to I recommend consulting the authors for help.

I’m afraid I cannot help. It is clear that the model you’ve presented in LaTeX is not compliant with the CVX ruleset. I have some ideas on how to handle the inequalities, but not the objective function. I would suggest connecting with the paper’s authors.

In fact, the abstract of the article specifically says their model is not convex. CVX is for convex programs only.