Hello, and a hearty welcome to the forum!!
trace(Q'*A*A'*Q) can be rewritten as square_pos(norm(A'*Q,'fro')) , which is convex. But maximizing that is not convex. Not squaring the ‘fro’ norm would be equivalent, and is convex, but maximizing it is not convex.
And the orthonormality constraint Q'*Q == eye(n) is non-convex.
So, except for the objective and constraint being non-convex, this would be a convex optimization problem.