What is the information plane theorem for an autoencoder neural network? Slide 8 (about 19 minutes into the video) of the Stanford Seminar - Information Theory of Deep Learning, Naftali Tishby has the following (rather informally stated) theorem.

Theorem (Information Plane) For large typical $\mathbf{X}$, the sample complexity of a DNN is completely determined by the encoder mutual information $\mathbf{I(X;T)}$, of the last hidden layer; the accuracy (generalization error) is determined by the decoder information, $\mathbf{I(T;Y)}$, of the last hidden layers.

I am having difficulty following what is meant by "the sample complexity is completely determined". What is the precise statement of this theorem?

The basic idea is to apply information theoratic principles to undertand the flow of information between layers. There are two important metrics in this regard.

*

*The entropy

*The Mutual information between the layers

So the Information bottleneck principle creates an information plane with x axis capturing the mutual information between X and Z (encoder) whereas Y  axis captures Mutual Information between Z and Y (Decoder). The basic finding based on experiments is that during the initial stages of training the Mutual information on both axis increases but after a certain point (called the bottleneck), the MI between X and Z starts to get less. This is called the compression phase. Intuitively this means that Z starts to only remember meaningful information from X and forget the rest.
Hope this helps
