That is neither convex nor concave. The product of variables ruins convexity. Note that trace_inv(X) is convex for an affine positive semidefintie argument, which is not the case for your expression.
Consider the scalar case. Let R = \alpha = 1. The expression is then 1/(1+P^2*Q^2); That is neither convex nor concave. For example, its Hessian at P = 1, Q = 0.5 has one positive eigenvalue and one negative eigenvalue.