Answers.Body: <p>Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just'  100K samples.  However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether they are useful or not, let alone get good values for coefficients).  For an application like this one, you would probably find most of the value from a few hundred words.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-14T17:23:48.803Z
Answers.LastActivityDate: 2019-03-14T17:23:48.803Z
Answers.UserDisplayName: Edward Dixon
Body: <p>Suppose we want to classify a review as good (<span class="math-container">$1$</span>) or bad (<span class="math-container">$0$</span>). We have a training data set of <span class="math-container">$10,000$</span> reviews. Also, suppose we have a vocabulary of <span class="math-container">$100,000$</span> words <span class="math-container">$w_1, \dots, w_{100,000}$</span>. So the data is a matrix of dimension <span class="math-container">$100,000 \times 10,000$</span>. Let's represent each of the words in the reviews using a bag-of-words approach over tf-idf values. Also, we normalize the rows such that they sum to <span class="math-container">$1$</span>.</p>
  <p>In a logistic regression approach, would we have <span class="math-container">$10,000$</span> different logistic regression models as follows:</p>
  <p><span class="math-container">$$ \log \left(\frac{p}{1-p} \right)_{1} = \beta_{0_{1}} + \beta_{1_{1}}w_{11} + \dots + \beta_{100,000_{1}}w_{100,000} \\ \vdots \\ \log \left(\frac{p}{1-p} \right)_{10,000} = \beta_{0_{10,000}} + \beta_{1_{10,000}}w_{11} + \dots + \beta_{100,000_{10,000}}w_{100,000}$$</span></p>
  <p>So are we estimating <span class="math-container">$100,000 \times 10,000$</span> coefficients?</p>
  
Tags: <natural-language-processing><weights><logistic-regression><binary-classification><bag-of-words>
Title: How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?
UserDisplayName: aiguy123
ThreadText: # How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?
  
  (Asked by: username_0 on 2019-03-13)
  
  Suppose we want to classify a review as good ($1$) or bad ($0$). We have a training data set of $10,000$ reviews. Also, suppose we have a vocabulary of $100,000$ words $w\_1, \dots, w\_{100,000}$. So the data is a matrix of dimension $100,000 \times 10,000$. Let's represent each of the words in the reviews using a bag-of-words approach over tf-idf values. Also, we normalize the rows such that they sum to $1$.
  
  In a logistic regression approach, would we have $10,000$ different logistic regression models as follows:
  
  $$ \log \left(\frac{p}{1-p} \right)\_{1} = \beta\_{0\_{1}} + \beta\_{1\_{1}}w\_{11} + \dots + \beta\_{100,000\_{1}}w\_{100,000} \\ \vdots \\ \log \left(\frac{p}{1-p} \right)\_{10,000} = \beta\_{0\_{10,000}} + \beta\_{1\_{10,000}}w\_{11} + \dots + \beta\_{100,000\_{10,000}}w\_{100,000}$$
  
  So are we estimating $100,000 \times 10,000$ coefficients?
  
  **Tags:** natural-language-processing, weights, logistic-regression, binary-classification, bag-of-words
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-14. Score: 2
  
  Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just' 100K samples. However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether they are useful or not, let alone get good values for coefficients). For an application like this one, you would probably find most of the value from a few hundred words.
  
  
  ---
  
  