That’s because this problem is not convex. The rules in the user guide are non-negotiable.

However, it is quasiconvex. That means, for fixedt>0, you can solve this feasibility problem:

cvx_begin sdp
variable X(n,n) symmetric
lambda_max(X) <= t * lambda_min(X)
X <= A' * X * A
X >= 0
cvx_end

Note that t is not a variable; that cannot be changed. You can, however, determine the optimal t using bisection.

Keep in mind, however, that this problem is homogeneous. So X=0 is an answer. Strict inequalities don’t work on CVX. You’ll have to do something about this; see Strict inequalities in the users’ guide. For this particular problem, my recommendation is to add this constraint: