-2/3*x^6+x^5 is convex for 0 <= x <= 1, but it is not convex for the entirety of its natural domain (-Inf <= x <= inf). I don’t know of a way of representing it in CVX. I am not ruling out that a clever forum reader might come up with a way, but I don’t think it is likely.

it’s not that CVX thinks it is concave. if -2/3*x^6+x^5 is entered directly, CVX concludes that it violates its rules, and it can’t determine its curvature (convex, concave, or neither). It also violates the rules for use of poly_env.

Presuming you actually want to maximize this function, it is a concave programming problem with compact constraint region. Therefore, there is a global maximum at an extreme, of which there are only two, namely the endpoints, 0 and 1. Therefore, whichever of 0 and 1 has the larger objective value is the global optimum. If you actually wished to minimize it, a local nonlinear solver such as FMINCON should find the global minimum, because the optimization problem would be convex (due to being constrained to [0,1]).