I found a way to solve the above problem using YALMIP. I came to know that, CVX with Mosek solver (academic license) cannot handle binary semidefinite variables. Below given is the mosek error displayed in matlab.
Mosek error: MSK_RES_ERR_GLOBAL_INV_CONIC_PROBLEM (The global optimizer can only be applied to problems without semidefinite variables.)
Whereas YALMIP can handle binary semidefinite variables. I have solved using the following code.
n1=5;n2=3;
A=rand(n1,n1)
x = diag(binvar(n1,1));
F = [sum(sum(x))==n2];
h = lambda_max(AxA’);
optimize(F,h);
sol = value(x)