Need Help with Convex Reformulation for Matrix Product in CVX Optimization

Nonconvex
2

As you said, this is a non-convex problem. Consider using YALMIP or some other tool which can handle nonlinear least squares.