\frac {k_1k_2x_1x_2} {k_1x_1+k_2x_2+N_0} is proved to be concave by calculating its Hessian martix. Its determinant can be written as -\frac {(k_1k_2x_1x_2-(k_1x_1+N_0)(k_2x_2+N_0))^2} {(k_1x_1+k_2x_2+N_0)^6} .

Therefore, the objective function is concave. I want to solve this problem with CVX, but CVX can not recognize this problem. I do not know how to rewrite this problem. If CVX does not work, is there any other way to solve this problem?

According to my calculations using MAPLE, the objective function is not concave. It is also the case that the argument of log is not concave.

Consider f(x1,x2) = x1*x2/(x1+x2+a) and h(x1,x2) = log(1 + f(x1,x2)) . These both have indefinite Hessians at x1 = 1, x2 = 1, a = 2; Hessian of f has eigenvalues 1/16 and -1/4. Hessian of h has eigenvalues 1/200 and -1/5.

Indeed, your determinant does equal the product of my eigenvalues, so is corect for at least those parameters. And that determinant is negative, which shows that there must be one positive and one negative eigenvalue, and hence that h is indefinite.