If you defined a new variable (variables?) consisting of F_{BB}F_{BB}^H, which would be declared as hemitian_semidefinite, and presuming \lambda_{i,j} \ge 0, then the argument of log_det would be affine henmitian semidefinite, so R would be the sum of log_det/(...)/log(2) But that would only be viable if F_{BB} is not needed by itself elsewhere in your problem.
Perhaps someone else has a better idea. Is there a determinantal identity which can be used?
But in any event, have you proven your optimization problem is convex?