I have to minimize the empirical risk of this modified loss function L*:
L*( t, y ) = L( t, y ) - L( t, -y ) where L( t, y ) = log(1 + exp( -ty )) is the logistic Loss.
Now if I do this in CVX I get the following:
Disciplined convex programming error:
Illegal operation: {convex} - {convex}
Is there any other way to write it ?
I’m doing this where t = w’x:
cvx_begin
variables w(p+1)
s=0;
for i=1:n
s=s + (log(1+exp( -y(i)*X(i,:)*w )) - log(1+exp( y(i)*X(i,:)*w )));
end
minimize (s)
cvx_end
I’ve also tried with the more compact matrix vector notation using the log sum exp but I get the same error.
Thanks.
Thanks, but I’ve already saw that and it is not helpful. I don’t want to do a Logistic regression.
I just want to know an alternative way to write the Objective function in such a way that does not produce a DCP error.
L*(t,y) simplifies to -t*y . Is that really what you want?
There are some positive constant that I have omitted and make no possible your simplification, which by the way I don’t see. Anyway I just want to know what I have to do when I have an error like that.
My simplification is valid for the problem you showed (try inputting some numbers, and you will see). If that is not the problem you wish to address, then please show us what that problem is.
And the FIRST step is to make sure that any objective you wish to minimize is a convex function of the optimization variables. My simplified formulation of the function you showed is convex, because it is actually linear (affine).
Ok, the problem is minimizing the expression at the end of page 3 of this article where the loss L(t,y) is the logistic loss.
(1 - rho_{-y}) and rho_{y} are just probabilities.
Thanks.
What specifically is the optimization problem you are trying to solve? I don’t see
L( t, y ) - L( t, -y )
where L( t, y ) = log(1 + exp( -t*y ))
anywhere in that paper. Are you sure your problem makes sense? Because I’m not.
The problem I want to solve is the empirical risk minimization of that modified Loss Function, which in the paper is at the end of page 3. Then as L( t, y ) the paper choose some common loss functions, one of them is the Logistic Loss which is at the end of page 4.
As I wrote before, I don’t see L*(t,y) anywhere in the paper. if you just wish to use logiistc loss function (regression), then see the link I provided previously. if you have some other specific optimization problem to solve, then please show it explicitly. If you need help in formulating an appropriate statistical fitting criterion, I suggest you post at https://stats.stackexchange.com/, and then come back here if you have a convex optimization problem you need help applying CVX to.
L*(t,y) is the formula at the end of page 3. The paper called l~ (small l tilde). I HAVE A SPECIFIC OPTIMIZATION problem TO SOLVE and it is minimizing the empirical risk of the modified loss function which is at the end of page 3 (Call in whatever way you want L*, l~). Is there a way to do it in CVX that does not result in a DCP error when I use the Logistic Loss for l( t , y)?
I can’t be more explicit than that.
Thanks.
That formula is NOT the same as your L*(t,y). That formula has \rho_{+y} and \rho_{-y}, neither of which are defined as far as I can see, although \rho_{+1} and \rho_{-1} are.
I showed you how to enter L*(t,y) as -t*y . The function in your code is just this, where X(i,:)*w is t.
If you have a different function to minimize, please prove it is convex. Note that numerical values of \rho_{+y} and \rho_{-y}, do bear on the convexity of the formula on the bottom of p.3 if a logistic loss is used.
ρ+y and ρ-y are two probabilities which are positive and less than one. So they don’t matter in the convexity of the formula since logistic loss is convex. By the way, they are ρ-1 or ρ+1 depending on the label y.
The 2nd derivative with respect to y of
a*log(1 + exp(-t*y)) - b*log(1 + exp(t* y))
evaluated at a = 0.2, b = 0.5, t = 1, y = 0.5 is -0.00705, so it is concave at that point. Hence my remarks in my preceding comment. Note that I am ignoring the denominator, which is positive, so doesn’t affect convexity status.
Do you understand that your objective function is NOT logistic loss?
I’ve never said that my objective function is the logistic loss. My objective function is the empirical risk of the modified loss (the modified loss is the formula at p.3). The logistic loss is the l(t, y) in the formula.
Having said that, from the paper alog(1 + exp(-ty)) - blog(1 + exp(t y)) should be convex.
Nevertheless thanks for your answers. You don’t need to answer again.