I have a Binary Matrix \bf{A} of size N\times N, where N=100
I have an optimization variable matrix \bf U of size N\times M, where M=300
The constraint I have is
{\bf U}(:,m)^T{\bf A}{\bf U}(:,m)==0,m=1,2,\cdots,M
where, {\bf U}(:,m) is the $m$th column vector of \bf U, i.e., {\bf U}(:,m) is os size N\times 1
The constraints
{\bf U}(:,m)^T{\bf A}{\bf U}(:,m)=0,m=1,2,\cdots,M
can be rewritten as
\sum_{i=1}^N \sum_{j=1}^N A(i,j) U(i,m)U(j,m)=0,m=1,2,\cdots,M.
Next, you can linearize each of the U(i,m)U(j,m) terms by introducing Z(i,j,m)=U(i,m)U(j,m) and adding the following constraints
Z(i,:,m) \leq U(i,m)\\Z(:,j,m) \leq U(j,m)\\Z(i,j,m) \geq U(i,m) + U(j,m) - 1
\sum_{i=1}^N \sum_{j=1}^N A(i,j) Z(i,j,m)=0,m=1,2,\cdots,M.