You can “move” a convex constraint the objective via penalty term, and have the objective remain convex. But when you try to move a non-convex constraint to the objective via penalty term, the objective will become non-convex. @jackfsuia pointed out the non-convexity, hence your dilemma.