Examine the definiteness of the Hessian, if it exists. or better yet, if you can, try proving it by applying the DCP rules, in which case if you succeed, you will know how you can enter it into CVX. if you can formulate the Hessian, it is often easy to show the function is not convex (or concave) by numerically evaluating its eigenvalues at various argument values. If the Hessian has both positive and negative eigenvalues, the function is neither convex nor concave, and not possible be used in CVX.