I have a Binary Matrix \bf{A} of size N\times N, where N=100

I have an optimization variable matrix \bf U of size N\times M, where M=300

The constraint I have is

{\bf U}(:,m)^T{\bf A}{\bf U}(:,m)==0,m=1,2,\cdots,M

where, {\bf U}(:,m) is the $m$th column vector of \bf U, i.e., {\bf U}(:,m) is os size N\times 1

The constraints

{\bf U}(:,m)^T{\bf A}{\bf U}(:,m)=0,m=1,2,\cdots,M

can be rewritten as

\sum_{i=1}^N \sum_{j=1}^N A(i,j) U(i,m)U(j,m)=0,m=1,2,\cdots,M.

Next, you can linearize each of the U(i,m)U(j,m) terms by introducing Z(i,j,m)=U(i,m)U(j,m) and adding the following constraints

Z(i,:,m) \leq U(i,m)\\Z(:,j,m) \leq U(j,m)\\Z(i,j,m) \geq U(i,m) + U(j,m) - 1

\sum_{i=1}^N \sum_{j=1}^N A(i,j) Z(i,j,m)=0,m=1,2,\cdots,M.