That is non-convex,due to multiplying different variables. E.g., the (1,2) element of X*X' is X(1)*X(2), which is neither convex nor concave.
Depending on what is done with X*X', it is possible that the optimization problem is convex, and can be formulated in such as way as to never explicitly formulate X*X'. Or maybe it can’t. The readers don’t know, because we don’t know what your overall problem,is.
Please very carefully read
which you have previously been instructed to read.