Answers.Body: <p>Apart from the <strong>entropy</strong> and the <strong>cross-entropy</strong>, which are widely used in deep learning and you seem to be aware of, there is also the <strong>Kullback-Leibler divergence</strong> (also known as <em>relative entropy</em>), which is widely used in the context of <a href="http://proceedings.mlr.press/v37/blundell15.html" rel="nofollow noreferrer"><em>variational Bayesian neural networks</em></a> and <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer"><em>variational auto-encoders</em></a>, given that it's often part of the loss function that is minimized, i.e. the <strong>Evidence Lower BOund</strong>, which is a proxy objective function for the KL divergence between the prior and posterior distributions (which actually corresponds to the <strong>minimum description length</strong> needed to encode the data: huh?). See <a href="https://ai.stackexchange.com/a/20574/2444">this answer</a> for more details. There is also the <strong>mutual information</strong>, which has also been used as a measure of <em>uncertainty</em> in the context of Bayesian neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-22T21:30:43.850Z
Answers.LastActivityDate: 2020-11-23T01:57:17.827Z
Answers.LastEditDate: 2020-11-23T01:57:17.827Z
Answers.UserDisplayName: nbro
Body: <p>How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical constructions.</p>
  <p>Note that, I'm aware that basic concepts such as entropy is used for training decision trees, and so on. I'm looking for applications which use slightly more advanced concepts from information theory, whatever they are.</p>
  
Tags: <machine-learning><deep-learning><information-theory>
Title: Applications of Information Theory in Machine Learning
UserDisplayName: SpiderRico
ThreadText: # Applications of Information Theory in Machine Learning
  
  (Asked by: username_0 on 2020-10-27)
  
  How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical constructions.
  
  Note that, I'm aware that basic concepts such as entropy is used for training decision trees, and so on. I'm looking for applications which use slightly more advanced concepts from information theory, whatever they are.
  
  **Tags:** machine-learning, deep-learning, information-theory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-22. Score: 2 | Accepted answer
  
  Apart from the **entropy** and the **cross-entropy**, which are widely used in deep learning and you seem to be aware of, there is also the **Kullback-Leibler divergence** (also known as *relative entropy*), which is widely used in the context of [*variational Bayesian neural networks*](http://proceedings.mlr.press/v37/blundell15.html) and [*variational auto-encoders*](https://arxiv.org/pdf/1312.6114.pdf), given that it's often part of the loss function that is minimized, i.e. the **Evidence Lower BOund**, which is a proxy objective function for the KL divergence between the prior and posterior distributions (which actually corresponds to the **minimum description length** needed to encode the data: huh?). See [this answer](https://ai.stackexchange.com/a/20574/2444) for more details. There is also the **mutual information**, which has also been used as a measure of *uncertainty* in the context of Bayesian neural networks.
  
  
  ---
  
  