Thanks for the reply!
In my case, I eventually want to form G = \left(A'\right)^T A' and minimize G's condition number. However, I understand this is a quasiconvex function of G and can’t be handled directly in CVX. I have a way to address this using the bisection method.
The focus of my question is can I formulate P as a variable in CVX? My intuitive idea is that I want to extract the rows of A that minimize the condition number of G. So P needs to be a matrix that is like a permutation matrix, but not square.
As for .', this is simply a habit of mine to use MATLAB transpose and not conjugate transpose.