Machine Learning, Information Theory, Deep Learning, Artificial Intelligence.
layers and retaining only the information most relevant to Y. This is the information bottleneck. Mutual Information I(X,Y) = H(X) — H(X|Y) H -> Entropy H(X) -> Entropy of X H(X|Y) -> Conditional Entropy of X given Y In other words, H(X|Y) signifies how much uncertainty is removed form X if Y is