How to eliminate quadratic form of -log(det( X*A*X^T + B))

Nonconvex
2

This is non-convex and non-concave even in the scalar case.

Consider the special case where A = B = 1, the expression is then -log(X^2+1). Its 2nd derivative at X = 0.5 is 0.96, and at X = 2 is -0.24. So it’s neither convex nor concave.

So my idea is you should first prove convexity (or concavity), as discussed in
Why isn't CVX accepting my model? READ THIS FIRST! .