Tl;dr, Neural networks are deterministic and sometimes even reversible, which causes Shannon information measures to degenerate. But information theory seems useful. How can we square this (if it's possible at all)? The attempts so far in the literature are unsatisfying.

Here is a conceptual question: what is the Right Way to think about information theoretic quantities in neural network contexts?

Example: I've been recently thinking about information bottleneck methods: given some data distribution _P(X, Y)_, it tries to find features _Z_ specified by _P(Z|X)_ that have nice properties like minimality (small _I(X;Z)_) and sufficiency (big _I(Z;Y)_).

But as pointed out in the literature several times, the fact that neural networks implement a deterministic map makes these information theoretic quantities degenerate:

• if Z is a deterministic map of X and they’re both continuous, then I(X;Z) is infinite.
  • Binning / Quantizing them does turn Z into a stochastic function [...]

---

Outline:

(02:28) Treat the weight as stochastic:

(04:23) Use something other than shannon information measures:

---

First published:

Source:

Narrated by TYPE III AUDIO.