Answers.Body: <p>It's much easier to deal with logarithms, as the relevant numbers are usually very small or very large. If you have a long exponential expression, it's hard to see the difference, but if you're looking at 4.3 vs 5.6, you can immediately see what's happening. And logarithms are a well-known (and well-understood) way of achieving this compression. You can easily interpret the difference, depending on the base of the logarithm used.</p>
  <p>Quite often the <span class="math-container">$log_2$</span> is used when you're dealing with entropy or information, as those are usually expressed in bits.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T07:31:08.577Z
Answers.LastActivityDate: 2021-06-14T07:31:08.577Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>I would like to add details to Oliver's answer.</p>
  <p>From the book &quot;Pattern Recognition and Machine Learning&quot; by Bishop (Section 1.2.5):</p>
  <blockquote>
  <p>In practice, it is more convenient to maximize the log of the
  likelihood function. Because the logarithm is monotonically increasing
  function of its argument, maximization of the log of a function is
  equivalent to maximization of the function itself. Taking the log not
  only simplifies the subsequent mathematical analysis, but it also
  helps numerically because the product of a large number of small
  probabilities can easily underflow the numerical precision of the
  computer, and this is resolved by computing instead the sum of the log
  probabilities.</p>
  </blockquote>
  <p>That is, <span class="math-container">$\log$</span> is monotonically increasing and hence preserves the order and the locations of the extrema. For instance, if <span class="math-container">$p(x) \geq p(y)$</span> then <span class="math-container">$\log\big(p(x)\big) \geq \log\big(p(y)\big)$</span> also holds. Therfore, maximizing likelihood is equivalent to maximizing log-likelihood.</p>
  <p>Furthermore, it is extremely useful when calculating joint probabilities since a product can be replaced by a sum:</p>
  <p><span class="math-container">$$
  \log \left(\prod_i P(x_i)\right) = \sum_i \log \left( P(x_i)\right)
  $$</span></p>
  <p>This also makes calculation numerically stable and it is much easier to take a derivative of a sum of logarithms rather than to take a derivative of a product.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T16:06:51.990Z
Answers.LastActivityDate: 2021-06-16T16:06:51.990Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>Term frequency and inverse document frequency are well-known terms in information retrieval.</p>
  <p>I am presenting the definitions for both from p:12,13 of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/6.pdf" rel="nofollow noreferrer">Vector Semantics and Embeddings</a></p>
  <p><strong>On term frequency</strong></p>
  <blockquote>
  <p>Term frequency is the frequency of the word <span class="math-container">$t$</span> in the term frequency
  document <span class="math-container">$d$</span>. We can just use the raw count as the term frequency:</p>
  <p><span class="math-container">$$tf_{t, d} = \text{count}(t, d)$$</span></p>
  <p>More commonly <strong>we squash the raw frequency a bit, by using the
  <span class="math-container">$\log_{10}$</span></strong> of the frequency instead. The intuition is that a word appearing 100 times in a document doesn’t make that word 100 times more likely to be relevant to the meaning of the document.</p>
  </blockquote>
  <p><strong>On inverse document frequency</strong></p>
  <blockquote>
  <p>The <span class="math-container">$\text{idf}$</span> is defined using the fraction <span class="math-container">$\dfrac{N}{df_t}$</span>, where <span class="math-container">$N$</span> is the total number of documents in the collection, and <span class="math-container">$\text{df}_t$</span> is the number of documents in which term <span class="math-container">$t$</span> occurs.......</p>
  <p>Because of the large number of documents in many collections,  <strong>this measure too is usually squashed with a log function</strong>. The resulting  definition  for  inverse document frequency (<span class="math-container">$\text{idf}$</span>) is thus</p>
  <p><span class="math-container">$$\text{idf}_t = \log_{10} \left(\dfrac{N}{df_t} \right)$$</span></p>
  </blockquote>
  <p>If we observe the bolded portion of the quotes, it is evident that the <span class="math-container">$\log$</span> function is used commonly. It is not only used in these two definitions. It has been across many definitions in the literature. For example: <a href="https://ai.stackexchange.com/questions/28220/product-of-probabilities-raised-to-own-powers-that-can-be-used-for-entropy-calcu">entropy</a>, mutual information, log-likelihood. So, I don't think squashing is the only purpose behind using the <span class="math-container">$\log$</span> function.</p>
  <p>Is there any reason for selecting the logarithm function for squashing? Are there any advantages for <span class="math-container">$\log$</span> compared to any other squash functions, if available?</p>
  
Tags: <natural-language-processing><definitions><books><tf-idf><logarithm>
Title: Why do we commonly use the $\log$ to squash frequencies?
UserDisplayName: hanugm
ThreadText: # Why do we commonly use the $\log$ to squash frequencies?
  
  (Asked by: username_0 on 2021-06-13)
  
  Term frequency and inverse document frequency are well-known terms in information retrieval.
  
  I am presenting the definitions for both from p:12,13 of [Vector Semantics and Embeddings](https://web.stanford.edu/%7Ejurafsky/slp3/6.pdf)
  
  **On term frequency**
  
  > Term frequency is the frequency of the word $t$ in the term frequency
  > document $d$. We can just use the raw count as the term frequency:
  >
  > $$tf\_{t, d} = \text{count}(t, d)$$
  >
  > More commonly **we squash the raw frequency a bit, by using the
  > $\log\_{10}$** of the frequency instead. The intuition is that a word appearing 100 times in a document doesn’t make that word 100 times more likely to be relevant to the meaning of the document.
  
  **On inverse document frequency**
  
  > The $\text{idf}$ is defined using the fraction $\dfrac{N}{df\_t}$, where $N$ is the total number of documents in the collection, and $\text{df}\_t$ is the number of documents in which term $t$ occurs.......
  >
  > Because of the large number of documents in many collections, **this measure too is usually squashed with a log function**. The resulting definition for inverse document frequency ($\text{idf}$) is thus
  >
  > $$\text{idf}\_t = \log\_{10} \left(\dfrac{N}{df\_t} \right)$$
  
  If we observe the bolded portion of the quotes, it is evident that the $\log$ function is used commonly. It is not only used in these two definitions. It has been across many definitions in the literature. For example: [entropy](https://ai.stackexchange.com/questions/28220/product-of-probabilities-raised-to-own-powers-that-can-be-used-for-entropy-calcu), mutual information, log-likelihood. So, I don't think squashing is the only purpose behind using the $\log$ function.
  
  Is there any reason for selecting the logarithm function for squashing? Are there any advantages for $\log$ compared to any other squash functions, if available?
  
  **Tags:** natural-language-processing, definitions, books, tf-idf, logarithm
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-06-14. Score: 2
  
  It's much easier to deal with logarithms, as the relevant numbers are usually very small or very large. If you have a long exponential expression, it's hard to see the difference, but if you're looking at 4.3 vs 5.6, you can immediately see what's happening. And logarithms are a well-known (and well-understood) way of achieving this compression. You can easily interpret the difference, depending on the base of the logarithm used.
  
  Quite often the $log\_2$ is used when you're dealing with entropy or information, as those are usually expressed in bits.
  
  
  ---
  
  ### Answer by username_2 on 2021-06-16. Score: 1
  
  I would like to add details to Oliver's answer.
  
  From the book "Pattern Recognition and Machine Learning" by Bishop (Section 1.2.5):
  
  > In practice, it is more convenient to maximize the log of the
  > likelihood function. Because the logarithm is monotonically increasing
  > function of its argument, maximization of the log of a function is
  > equivalent to maximization of the function itself. Taking the log not
  > only simplifies the subsequent mathematical analysis, but it also
  > helps numerically because the product of a large number of small
  > probabilities can easily underflow the numerical precision of the
  > computer, and this is resolved by computing instead the sum of the log
  > probabilities.
  
  That is, $\log$ is monotonically increasing and hence preserves the order and the locations of the extrema. For instance, if $p(x) \geq p(y)$ then $\log\big(p(x)\big) \geq \log\big(p(y)\big)$ also holds. Therfore, maximizing likelihood is equivalent to maximizing log-likelihood.
  
  Furthermore, it is extremely useful when calculating joint probabilities since a product can be replaced by a sum:
  
  $$
  \log \left(\prod\_i P(x\_i)\right) = \sum\_i \log \left( P(x\_i)\right)
  $$
  
  This also makes calculation numerically stable and it is much easier to take a derivative of a sum of logarithms rather than to take a derivative of a product.
  
  
  ---
  
  