Information Bottleneck Analysis by a Conditional Mutual Information Bound

Task-nuisance decomposition describes why the information bottleneck loss I(z; x) - beta I(z; y) is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, I(z; n) can be decreased by reducing I (z; x) since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information I(z; x vertical bar y) provides an alternative upper bound for I(z; n). This bound is applicable even if z is not a sufficient representation of x, that is, I(z; y) not equal I(x; y). We used mutual information neural estimation (MINE) to estimate I (z; x vertical bar y). Experiments demonstrated that I(z; x vertical bar y) is smaller than I(z; x) for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when I(z; x vertical bar y) is used instead of I(z; x).

Automated summary

Task-nuisance decomposition explains why the information bottleneck loss is a suitable objective for supervised learning. By demonstrating that conditional mutual information provides an alternative upper bound for I(z;n), even if z is not a sufficient representation of x, we extend this framework.