Matlab OUT OF MEMORY error on optimization problem

Mosek is very clearly warning you about the value 2e8… Is that the value of param.mu you used? That apparent;y makes the problem excxessivley ill-conditioned to handle reliably.

See my posts in Prescribing tolerance for positivedefiniteness for how to get CVX to use the Mosek version you installed instead of the old version, 8.0.0.80, provided in the CVX installation. Perhaps the latest 8.1 or 9.0 beta version will be able to better deal with your numerically challenging problemr than 8.0.0.80. But in any event, try to keep the value of param.mu smaller.