I want to maximize objective function sum(x./(a*x+b)), where a and b are positive vectors, and x is the decision variable. In addition, M>0. The objective function is concave, since its second derivate is negative. However, I do not know how to formulate it in CVX.

cvx_begin quiet
variable x(N);
expression fx(N);

for i=1:N
fx(i)=x(i)*inv_pos(a(i)*x(i)+b(i));
end
maximize(sum(fx));
subject to x>=0;
sum(x)==M;

cvx_end

I realize that it is not allowed for the product of an affine (x) and a convex function (1/(a*x+b)) based on the rule. But I have no idea on how to transform it. Any help is appreciated.

Consider the simple case, f(x) = x/(x-1) for x a scalar. For x > 1, the 2nd derivative of f is positive, and for x < 1, it is negative. For f(x) = x/(x+1), for the positive portion of the denominator, the 2nd derivative is negative, and negative on the other side.

So, as mcg commented in reply to your original question, your function is neither convex nor concave, and so can not be handled in CVX.

There should be constraints x>=0 and sum(x)==M>0. Vectors a and b are positive. In this case, the second derivate of the function is concave. The question is how to write the statements in CVX.

Even if the constraints helped—and they do not—CVX does not have the luxury of being able to consider the global constraint set when determining convexity/concavity. All expressions must be convex or concave independently.