Answers.Body: <p>The paper that appears to have introduced the <em>term</em> &quot;softmax&quot; is <a href="https://papers.nips.cc/paper/195-training-stochastic-model-recognition-algorithms-as-networks-can-lead-to-maximum-mutual-information-estimation-of-parameters.pdf" rel="noreferrer">Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters</a> (1989, NIPS) by John S. Bridle.</p>
  <p>As a side note, the <a href="https://en.wikipedia.org/wiki/Softmax_function" rel="noreferrer">softmax function</a> (with base <span class="math-container">$b = e^{-\beta}$</span>)</p>
  <p><span class="math-container">$$\sigma (\mathbf {z} )_{i}={\frac {e^{-\beta z_{i}}}{\sum _{j=1}^{K}e^{-\beta z_{j}}}}{\text{ for }}i=1,\dotsc ,K {\text{ and }}\mathbf {z} =(z_{1},\dotsc ,z_{K})\in \mathbb {R} ^{K}$$</span></p>
  <p>is very similar to the <a href="https://en.wikipedia.org/wiki/Boltzmann_distribution" rel="noreferrer">Boltzmann (or Gibbs) distribution</a></p>
  <p><span class="math-container">$$
  p_i=\frac{e^{- {\varepsilon}_i / k T}}{\sum_{j=1}^{M}{e^{- {\varepsilon}_j / k T}}}
  $$</span></p>
  <p>which was formulated by Ludwig Boltzmann in 1868, so the idea and formulation of the softmax function is quite old.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T00:37:40.403Z
Answers.LastActivityDate: 2020-07-13T22:29:39.100Z
Answers.LastEditDate: 2020-07-13T22:29:39.100Z
Answers.UserDisplayName: nbro
Body: <p>Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned anywhere. So, which paper introduced this term?</p>
  
Tags: <neural-networks><deep-learning><classification><history><softmax>
Title: Which paper introduced the term "softmax"?
UserDisplayName: nbro
ThreadText: # Which paper introduced the term "softmax"?
  
  (Asked by: username_0 on 2020-07-10)
  
  Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned anywhere. So, which paper introduced this term?
  
  **Tags:** neural-networks, deep-learning, classification, history, softmax
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-07-10. Score: 7 | Accepted answer
  
  The paper that appears to have introduced the *term* "softmax" is [Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters](https://papers.nips.cc/paper/195-training-stochastic-model-recognition-algorithms-as-networks-can-lead-to-maximum-mutual-information-estimation-of-parameters.pdf) (1989, NIPS) by John S. Bridle.
  
  As a side note, the [softmax function](https://en.wikipedia.org/wiki/Softmax_function) (with base $b = e^{-\beta}$)
  
  $$\sigma (\mathbf {z} )\_{i}={\frac {e^{-\beta z\_{i}}}{\sum \_{j=1}^{K}e^{-\beta z\_{j}}}}{\text{ for }}i=1,\dotsc ,K {\text{ and }}\mathbf {z} =(z\_{1},\dotsc ,z\_{K})\in \mathbb {R} ^{K}$$
  
  is very similar to the [Boltzmann (or Gibbs) distribution](https://en.wikipedia.org/wiki/Boltzmann_distribution)
  
  $$
  p\_i=\frac{e^{- {\varepsilon}\_i / k T}}{\sum\_{j=1}^{M}{e^{- {\varepsilon}\_j / k T}}}
  $$
  
  which was formulated by Ludwig Boltzmann in 1868, so the idea and formulation of the softmax function is quite old.
  
  
  ---
  
  