My optimization problem is to maximize the minimum rate of the user under the transmit power constraint. I put the transmit power very small, for example -10dBm, the objective function value is convergent. However, when I amplified the transmission power, the value of the objective function increased slowly, and the result was failed. I’ve used different solvers, mosek/sedumi, and this problem comes up. How to solve this problem?

Remove cvx_quiet, and show us all the CVX and (Mosek) solver output for both the case which worked and the case which failed. It would be better if you can also show is the input data and make them into complete reproducible examples.

Perhaps you can improve things numerically by not squaring the LHS of the constraint, and instead taking the square root of the RHS.

Yes, the outputs and subsequent inputs of SCA can get more and more extreme, until at some point the solver fails. Of course, by that point, the algorithm is probably not converging to anything, let alone a global or even local optimum of the original problem.