Actually, there&#x27;s been a few papers showing solid theoretical progress on understanding what structure in data deep networks are learning to represent.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;gregversteeg&#x2F;CorEx" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;gregversteeg&#x2F;CorEx</a><p><a href="http:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1406.1222" rel="nofollow">http:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1406.1222</a><p><a href="http:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1410.7404" rel="nofollow">http:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1410.7404</a><p>Disclaimer: I reinvented the idea about a month or two ago, and ran into the existing papers when googling for joint entropy estimators after making some interesting graphs with hierarchical probabilistic programs.  I only wish I could&#x27;ve been the first!