U has a couple of (slightly) negative eigenvalues, so quad-_form will not accept this even if you fix the vec error. You need to adjust U in some manner.
As for the vec error, read Error "Undefined function 'vec' for input arguments of type 'double' Note that vec is called by quad_form.
An alternative workardound to the vec issue is to explicitly form the quadratic form as v'*U*v.