Error: Invalid quadratic form(s): not a square

A=[0 0 1 0;0 0 0 1;-1 1 0 0; 1 -1 0 0];
B=[0;0;1;0];
D=[0 0; 0 0; 1 0; 0 1];
H=[1 -1 0 0];
C1=[1 0 0 0;0 1 0 0];
C2=[0 0 1 0;0 0 0 1];
F=[0;0;0.2;0.2];
Y1=eye(2,2);
Y2=eye(1,1);
Z1=zeros(2,1);
Z2=zeros(1,2);
Y3=eye(4,4);
cvx_begin
variable P(4,4)  semidefinite;
variable Q(4,4)  semidefinite;
variable al ;
variable mu1;
variable mu2;
variable eps1;
variable eps2;
minimize (trace (C2*P*C2'))
subject to
[A*P+P*A'+al*P+mu1*B*B' D P*H';D' -al*Y1 Z1; H*P Z2 -eps1*Y2]<=0
[Q*A+A'*Q+al*Q-mu2*C1'*C1+eps2*H'*H Q*D Q*F;D'*Q -al*Y1 Z1; F'*Q Z2 -eps2*Y2]<=0
[P eye(4,4);eye(4,4) Q]>=0
cvx_end

error

 Error using cvx/times (line 262)
 Disciplined convex programming error:
    Invalid quadratic form(s): not a square.

Error in cvx/mtimes (line 36)
    z = feval( oper, x, y );

I recently started to learn it, please help me

There are a couple of problems.

You have declared al to be a variable, and it appears in a multiplicative way with the variables P and Q, which is a violation of CVX’s Disciplined Convex Programming rules, and indeed is non-convex… Your problem can be solved if al is set equal to some numerical value, rather than being declared a CVX variable.

Indeed, I believe your problem is quasi-convex (I stand ready to be corrected if it’s not), and therefore can be solved by bisection - see “Quasiconvex optimization” (section 4.2.5), and in particular, the bisection solution algoirthm on pp. 145-146 of http://stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf ,

Your other problem is easy to fix. As entered, your constraints will be interpreted elementwise. You obviously intend for the constraints to be LMIs. The easiest way to fix is that is to invoke CVX’s SDP mode by using

cvx_begin sdp

rather than

cvx_begin

That said, your last constraint is redundant (does no harm, but is not necessary), since it can only be satisfied as an LMI if P and Q are both positive semi-definite; but you already declared them to be so in your variable declarations.

If you make those changes, the program will run. Presuming that your problem is indeed quasi-convex, you should be able to solve it by using the aforementioned bisection algorithm, calling CVX to solve the feasibility problems dictated by the bisection algorithm…