How to minimize my log_det function in convex programming

I think, we can neglect the L0 norm constraint, because it can be relaxed by L1 norm. The difficulty to me is how to formulate R_p matrix as a linear function of p vector, then it maybe accepted by CVX.

This is not convex even if you relax the binary requirement on p.

Yes, I agree. So how to formulate R_p matrix as a linear function of p vector, such that it maybe accepted by CVX?

There is no way to make CVX solve this problem.

So do you have any suggestion for me to solve my problem? thanks for your time.

If M is small enough, you could you use brute force evaluation of the objective function for all possible values of p.

Otherwise, perhaps use a general purpose mixed integer nonlinear solver and recognize/accept that you might not get the global optimum, or try posting at https://www.or-exchange.org/ or http://scicomp.stackexchange.com/ .