How do I solve a problem with the Trace of quadratic form

Apply the same transformation as in this answer you “liked” Help me for this objective function .

maximize(-square_pos(norm(sqrtm(A)*X,'fro')))

The argmax is preserved if instead you use maximize(-norm(sqrtm(A)*X,'fro')) which avoids unnecessary squarng, and might be better behaved numerically.