# Non-scalar quadratic forms with non-PSD and non-Hermitian coefficient matrices

I need to implement the expression obj = \mbox{tr}(AXB'Y'A'X'BY) in CVX, where A and B are known complex matrices and X and Y are unknown complex matrices, which I intend to optimize X and Y with alternating optimization methods. As CVX only accepts scalar quadratic forms, the only way that comes to my mind, is to rewrite obj in a Frobenius norm form. But the order of placement of the matrices is not the way to be rewritten like that.

The other trick is to rewrite obj in each iteration, considering the optimization in that iteration is respect to X or Y. Thus, for optimization with respect to X we have

obj = \mbox{tr}((BYA)X(AYB)'X') = \mbox{tr}(CXDX')

and for the optimization with respect to Y we have

obj = \mbox{tr}((AXB')Y'(B'XA)'Y) = \mbox{tr}(EY'FY).

But the problem here is that neither X nor Y are PSD and Hermitian, and I don’t think that there would be such a matrix decomposition approach, which be able to decompose a non-PSD, non-hermitian matrix R into something like R = TT', so we would be able to rewrite obj in obj = ||P||^2_F format. Maybe it’s good to say, that matrices A and B are both Hermitian and can be decomposed as R = TT', but having X and Y non-PSD and non-hermitian, makes it challenging. For those who haven’t got the problem, I should say, that, if for example C and D were Hermitian matrices, we could rewrite optimization with respect to X as follows.

obj = \mbox{tr}(C^{1/2}(C^{1/2})'XD^{1/2}(D^{1/2})'X') = ||(C^{1/2})'XD^{1/2}||^2_F,

knowing that,

||M||^2_F = \mbox{tr}(MM'),

and this kind of Frobenius norm follows DCP rules, while the original format of obj is not accepted.

So, can someone please tell what I can do with the non-PSD and non-Hermitian properties of the referened matrices, in order to convert obj into a Frobenius norm?

Thanks,