Log of sigmoid function

3

It might be a bit surprising to CVX power users that log(1/(1+exp(-z))) is accepted by CVX. After all, it does not seem to obey the disciplined convex programming ruleset!

But in fact, in order to support geometric programming, CVX has a lesser known set of rules governing log convexity as well. The definition of log convexity is this: if a function is positive and its logarithm is convex, then it is log-convex. There are equivalent definitions of log-affine and log-concave functions as well.

We don’t publish the log-convexity rules explicitly. But if you’re a power user you might be interested in them. First, we have the following rules that take you back and forth between the “standard” convexity world and the log-convexity world:

  • exp(*affine*) is log-affine; exp(*convex*) is log-convex; exp(*concave*) is log-concave.
  • log(*log-affine*) is affine; log(*log-convex*) is convex; log(*log-concave*) is concave.
  • Log-affine expressions are also log-concave and log-convex.
  • Positive constants are log-affine, log-concave, and log-convex.

Given those basics, you can actually take most of the basic DCP rules and map them to log-convexity:

  • Products: products of {log-affine,log-convex,log-concave} expressions are {log-affine,log-convex,log-concave}. This is the log-convexity equivalent of DCP’s sum rules.
  • Positive exponents: A {log-affine,log-convex,log-concave} expression raised to a positive constant power is {log-affine,log-convex,log-concave}. This is the log-convexity equivalent of DCP’s positive scaling rules.
  • Negative exponents: A {log-affine,log-convex,log-concave} expression raised to a negative constant power is {log-affine,log-concave,log-convex}. This is the log-convexity equivalent of DCP’s negative scaling rules.
  • Ratios: The ratio of a {log-convex,log-concave,log-affine} and a {log-concave,log-convex,log-affine} expression is {log-convex,log-concave,log-affine}. This is the equivalent of differences in DCP. So, not surprisingly, taking the ratio of two log-convex expressions is invalid, because this would be equivalent to subtracting one convex expression from another.

There is one rule for log-convexity, however, that does not have a DCP analog. It is this:

  • Sums of log-affine and/or log-convex expressions are log-convex. Sums of log-concave expressions are not permitted. Note that even if every term of the sum is log-affine, the resulting sum is log-convex.

Confusing? Well, yeah, it is. But let’s go through log(1/(1+exp(-z))) to see why CVX accepts it:

  • Both 1 and exp(-z) are log-affine.
  • The sum 1+exp(-z) is therefore log-convex.
  • The ratio 1/(1+exp(-z)) is log-concave. Alternatively, it is equivalent to (1+exp(-z))^(-1), which is log-concave by the negative exponent rule.
  • Therefore, log(1/(1+exp(-z))) is concave.
3 Likes