Everything about your problem, both the objective and the constraint, are highly non-convex.
Expanding on @Michal_Adamaszek 's post:
You could consiider using YALMIP.
Maybe you can relax this formula using SDP and then using the Dinkelbach algorithm to solve it?