xlog(1+ ay/x) can be expressed
-rel_entr(x,x+ay)
xlog((by+x)/(cy+x)) can be expressed
1/(1-a/c)*(rel_entr(a*y+x,c*y+x)+a/c* rel_entr(c*y+x,a*y+x)
therefore, the problem can be expressed
1/(1-a/c)*(rel_entr(a*y+x,c*y+x)+a/c* rel_entr(c*y+x,a*y+x) - rel_entr(x,x+ay)?
“Correct” or not, you’re not going to be able to subtract two nonnegative rel_entr terms. That is {convex expression} - {convex expression}, which is not allowed by CVX’s DCP rules.
The first term of the original expression is concave. The 2nd term is not concave; perhaps it is convex. The overall expression is not convex; perhaps it is concave. Therefore, if it is concave, you will have to formulate a single CVX expression for the overall expression,
But your first step is to prove this expression is concave (as opposed to neither concave nor convex).
Then, what method should I use to solve this problem?
If you want to solve a non-convex optimization problem, use a non-convex local or global optimization solver, for instance under YALMIP.