Answers.Body: <p>The multinomial model generalizing the binomial model is a probabilistic model primarily used for categorical data where each observation belongs to one of <span class="math-container">$k&gt;2$</span> discrete categories, which can be applied in tasks like text classification, topic modeling. A common such model before the era of LLMs is the <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes" rel="nofollow noreferrer">Multinomial Naive Bayes classifier (MNB)</a> where each document represented as a <span class="math-container">$k$</span>-dimensional Bag of Words (BOW) feature is assumed to follow the <a href="https://en.wikipedia.org/wiki/Multinomial_distribution" rel="nofollow noreferrer">Multinomial distribution</a>, each observed word is a trial, the vocabulary corresponds to the <span class="math-container">$k$</span> possible categories, and the conditional probabilities <span class="math-container">$p_i$</span> of each observed word given a class label are mutually independent. Thus it's better at capturing term frequency information compared to other models such as Bernoulli model where only presence or absence of words are taken into account, which can be critical for classification accuracy.</p>
  <p>Suppose we want to classify an email with BOW representation whose vocabulary is [&quot;win&quot;, &quot;money&quot;, &quot;free&quot;] (only 3 discrete categories for any word observation) and classes labels are Spam and Not-Spam. Next from training data we know the estimated conditional probabilities of each word in the vocabulary given email class, say, <span class="math-container">$$P(win∣Spam)=0.3,P(money∣Spam)=0.5,P(free∣Spam)=0.2$$</span>
  <span class="math-container">$$P(win∣Not-Spam)=0.1,P(money∣Not-Spam)=0.2,P(free∣Not-Spam)=0.7$$</span></p>
  <p>Now we test an incoming email as BOW feature <span class="math-container">$X=(win: 2, money: 1, free: 1)$</span>. First we use the Multinomial distribution formula to compute likelihood for each class label as follow:
  <span class="math-container">$$P(X∣Spam)=\frac{4!}{2!1!1!}(0.3)^2(0.5)^1(0.2)^1$$</span>
  <span class="math-container">$$P(X∣Not-Spam)=\frac{4!}{2!1!1!}(0.1)^2(0.2)^1(0.7)^1$$</span></p>
  <p>Finally we combine above computed likelihoods with class priors to determine which class the email most likely belongs to according to the Bayes theorem.</p>
  <p>Since MNB is a very popular text classification ML model, <code>sklearn</code> has a <code>MultinomialNB</code> class in its <code>naive_bayes</code> module. You can try below sample Multinomial model.</p>
  <pre><code>import pandas as pd
  from sklearn.feature_extraction.text import CountVectorizer
  from sklearn.model_selection import train_test_split
  from sklearn.naive_bayes import MultinomialNB
  from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
  
  # Expanded example dataset (add more rows to increase model training potential)
  data = {
      'text': [
          'Win a free iPhone now',
          'Congratulations, you won a prize',
          'Meeting at 3 PM today',
          'Lunch at the usual spot',
          'Claim your free gift now',
          'Free money for you today',
          'Get your prize now',
          'This is not spam, just a meeting reminder',
          'Check out the new product',
          'Free vacation offer, click here'
      ],
      'label': ['spam', 'spam', 'not-spam', 'not-spam', 'spam', 'spam', 'spam', 'not-spam', 'not-spam', 'spam']
  }
  
  df = pd.DataFrame(data)
  df['label'] = df['label'].map({'not-spam': 0, 'spam': 1})
  
  # Split data into training and testing sets
  X_train, X_test, y_train, y_test = train_test_split(
      df['text'], df['label'], test_size=0.2, random_state=42
  )
  
  # Convert text to a Bag-of-Words representation
  vectorizer = CountVectorizer()
  X_train_bow = vectorizer.fit_transform(X_train)
  X_test_bow = vectorizer.transform(X_test)
  
  # Initialize and train the Multinomial Naive Bayes model
  model = MultinomialNB(class_prior=[0.5, 0.5])
  model.fit(X_train_bow, y_train)
  
  # Make predictions on the test set
  y_pred = model.predict(X_test_bow)
  
  # Evaluate the model
  print(&quot;Accuracy:&quot;, accuracy_score(y_test, y_pred))
  print(&quot;Confusion Matrix:\n&quot;, confusion_matrix(y_test, y_pred))
  print(&quot;Classification Report:\n&quot;, classification_report(y_test, y_pred))
  
  # Example new messages
  new_messages = [
      'Win cash prizes now',
      'Are we meeting tomorrow?'
  ]
  
  # Vectorize new messages
  new_messages_bow = vectorizer.transform(new_messages)
  
  # Predict
  predictions = model.predict(new_messages_bow)
  
  # Display predictions for new messages
  for msg, pred in zip(new_messages, predictions):
      label = 'Spam' if pred == 1 else 'Not-Spam'
      print(f&quot;Message: '{msg}' =&gt; Prediction: {label}&quot;)
  
  </code></pre>
  <p>Output is expected as:</p>
  <pre><code>Accuracy: 1.0
  Confusion Matrix:
   [[1 0]
   [0 1]]
  Classification Report:
                 precision    recall  f1-score   support
  
             0       1.00      1.00      1.00         1
             1       1.00      1.00      1.00         1
  
      accuracy                           1.00         2
     macro avg       1.00      1.00      1.00         2
  weighted avg       1.00      1.00      1.00         2
  
  Message: 'Win cash prizes now' =&gt; Prediction: Spam
  Message: 'Are we meeting tomorrow?' =&gt; Prediction: Not-Spam
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-25T06:31:50.073Z
Answers.LastActivityDate: 2024-12-25T06:31:50.073Z
Answers.UserDisplayName: cinch
Answers.Body: <p>A multinomial model is the statistical terminology for counting n repeated draws according to a softmax classifier (regressor to be precise).</p>
  <p>Update after @David comment:
  Categorical and multinomial distribution are used interchangeably ( technically incorrect though), as often in practice what is unknown is the probability vector p, but not the sample size n.  Also for ML models at inference time n is usually known (e.g. n=1 for predicting the next sample).  So the learning problem is to map features X to a probability vector p over K categories, e.g. through a softmax last layer.</p>
  <p>This is analogous to how when dealing with binomial outcomes (counts), practitioners refer to binomial vs Bernoulli problem setup interchangeably as also here often n is known / fixed, whereas the probability of p(y=1|X) is unknown and has to be learned through e.g. a sigmoid (logistic) regression setup.</p>
  <p>See also
  <a href="https://discourse.mc-stan.org/t/multilevel-categorical-multinomial-model-terms-and-priors/5294" rel="nofollow noreferrer">https://discourse.mc-stan.org/t/multilevel-categorical-multinomial-model-terms-and-priors/5294</a></p>
  <p><a href="https://beginningwithml.wordpress.com/2018/06/22/3-4-softmax-regression/" rel="nofollow noreferrer">https://beginningwithml.wordpress.com/2018/06/22/3-4-softmax-regression/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-30T16:36:54.107Z
Answers.Text: Not quite. Softmax would be the same as a categorical distribution. Multinomial is a generalisation of the binomial distribution.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-30T20:22:26.730Z
Answers.Text: @David in ml context of prediction models at inference time ppl are interested in the prediction of a known "n" ( e.g. n=1).  So yes I incorrectly put softmax as the equivalence to multinomial, but in (ml/predictuve) practice ppl usually mean the instance level inference where n Is assumed to be known/given (n=1 or sample size) and all that had to be learned is the probability vector just like for binomial models it's common to learn "p" , not also n.  Will update comment to clarify and add references
Answers.UserDisplayName: Georg M. Goerg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-30T14:16:26.400Z
Answers.LastActivityDate: 2023-11-30T20:30:20.887Z
Answers.LastEditDate: 2023-11-30T20:30:20.887Z
Answers.UserDisplayName: Georg M. Goerg
Body: <p>What is the mathematical definition of &quot;multinomial model&quot; in machine learning?
  I will be happy for a good definition plus an example.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-29T19:48:15.853Z
Comments.Text: Are you asking in neural network or Random Forest?
Comments.UserDisplayName: Cloud Cho
Tags: <machine-learning><statistics>
Title: What is a "multinomial model" in machine learning?
UserDisplayName: DSPinfinity
ThreadText: # What is a "multinomial model" in machine learning?
  
  (Asked by: username_0 on 2023-11-29)
  
  What is the mathematical definition of "multinomial model" in machine learning?
  I will be happy for a good definition plus an example.
  
  **Tags:** machine-learning, statistics
  > **Comments:**
  > **username_1**: Are you asking in neural network or Random Forest?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-12-25. Score: 0
  
  The multinomial model generalizing the binomial model is a probabilistic model primarily used for categorical data where each observation belongs to one of $k>2$ discrete categories, which can be applied in tasks like text classification, topic modeling. A common such model before the era of LLMs is the Multinomial Naive Bayes classifier (MNB) where each document represented as a $k$-dimensional Bag of Words (BOW) feature is assumed to follow the Multinomial distribution, each observed word is a trial, the vocabulary corresponds to the $k$ possible categories, and the conditional probabilities $p_i$ of each observed word given a class label are mutually independent. Thus it's better at capturing term frequency information compared to other models such as Bernoulli model where only presence or absence of words are taken into account, which can be critical for classification accuracy.
  Suppose we want to classify an email with BOW representation whose vocabulary is ["win", "money", "free"] (only 3 discrete categories for any word observation) and classes labels are Spam and Not-Spam. Next from training data we know the estimated conditional probabilities of each word in the vocabulary given email class, say, $$P(win∣Spam)=0.3,P(money∣Spam)=0.5,P(free∣Spam)=0.2$$
  $$P(win∣Not-Spam)=0.1,P(money∣Not-Spam)=0.2,P(free∣Not-Spam)=0.7$$
  Now we test an incoming email as BOW feature $X=(win: 2, money: 1, free: 1)$. First we use the Multinomial distribution formula to compute likelihood for each class label as follow:
  $$P(X∣Spam)=\frac{4!}{2!1!1!}(0.3)^2(0.5)^1(0.2)^1$$
  $$P(X∣Not-Spam)=\frac{4!}{2!1!1!}(0.1)^2(0.2)^1(0.7)^1$$
  Finally we combine above computed likelihoods with class priors to determine which class the email most likely belongs to according to the Bayes theorem.
  Since MNB is a very popular text classification ML model, sklearn has a MultinomialNB class in its naive_bayes module. You can try below sample Multinomial model.
  import pandas as pd
  from sklearn.feature_extraction.text import CountVectorizer
  from sklearn.model_selection import train_test_split
  from sklearn.naive_bayes import MultinomialNB
  from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
  
  # Expanded example dataset (add more rows to increase model training potential)
  data = {
      'text': [
          'Win a free iPhone now',
          'Congratulations, you won a prize',
          'Meeting at 3 PM today',
          'Lunch at the usual spot',
          'Claim your free gift now',
          'Free money for you today',
          'Get your prize now',
          'This is not spam, just a meeting reminder',
          'Check out the new product',
          'Free vacation offer, click here'
      ],
      'label': ['spam', 'spam', 'not-spam', 'not-spam', 'spam', 'spam', 'spam', 'not-spam', 'not-spam', 'spam']
  }
  
  df = pd.DataFrame(data)
  df['label'] = df['label'].map({'not-spam': 0, 'spam': 1})
  
  # Split data into training and testing sets
  X_train, X_test, y_train, y_test = train_test_split(
      df['text'], df['label'], test_size=0.2, random_state=42
  )
  
  # Convert text to a Bag-of-Words representation
  vectorizer = CountVectorizer()
  X_train_bow = vectorizer.fit_transform(X_train)
  X_test_bow = vectorizer.transform(X_test)
  
  # Initialize and train the Multinomial Naive Bayes model
  model = MultinomialNB(class_prior=[0.5, 0.5])
  model.fit(X_train_bow, y_train)
  
  # Make predictions on the test set
  y_pred = model.predict(X_test_bow)
  
  # Evaluate the model
  print("Accuracy:", accuracy_score(y_test, y_pred))
  print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
  print("Classification Report:\n", classification_report(y_test, y_pred))
  
  # Example new messages
  new_messages = [
      'Win cash prizes now',
      'Are we meeting tomorrow?'
  ]
  
  # Vectorize new messages
  new_messages_bow = vectorizer.transform(new_messages)
  
  # Predict
  predictions = model.predict(new_messages_bow)
  
  # Display predictions for new messages
  for msg, pred in zip(new_messages, predictions):
      label = 'Spam' if pred == 1 else 'Not-Spam'
      print(f"Message: '{msg}' => Prediction: {label}")
  
  
  Output is expected as:
  Accuracy: 1.0
  Confusion Matrix:
   [[1 0]
   [0 1]]
  Classification Report:
                 precision    recall  f1-score   support
  
             0       1.00      1.00      1.00         1
             1       1.00      1.00      1.00         1
  
      accuracy                           1.00         2
     macro avg       1.00      1.00      1.00         2
  weighted avg       1.00      1.00      1.00         2
  
  Message: 'Win cash prizes now' => Prediction: Spam
  Message: 'Are we meeting tomorrow?' => Prediction: Not-Spam
  
  
  ---
  
  ### Answer by username_3 on 2023-11-30. Score: 0
  
  A multinomial model is the statistical terminology for counting n repeated draws according to a softmax classifier (regressor to be precise).
  
  Update after @David comment:
  Categorical and multinomial distribution are used interchangeably ( technically incorrect though), as often in practice what is unknown is the probability vector p, but not the sample size n. Also for ML models at inference time n is usually known (e.g. n=1 for predicting the next sample). So the learning problem is to map features X to a probability vector p over K categories, e.g. through a softmax last layer.
  
  This is analogous to how when dealing with binomial outcomes (counts), practitioners refer to binomial vs Bernoulli problem setup interchangeably as also here often n is known / fixed, whereas the probability of p(y=1|X) is unknown and has to be learned through e.g. a sigmoid (logistic) regression setup.
  
  See also
  <https://discourse.mc-stan.org/t/multilevel-categorical-multinomial-model-terms-and-priors/5294>
  
  <https://beginningwithml.wordpress.com/2018/06/22/3-4-softmax-regression/>
  
  > **Comments:**
  > **username_4**: Not quite. Softmax would be the same as a categorical distribution. Multinomial is a generalisation of the binomial distribution.  
  > **username_3**: @David in ml context of prediction models at inference time ppl are interested in the prediction of a known "n" ( e.g. n=1).  So yes I incorrectly put softmax as the equivalence to multinomial, but in (ml/predictuve) practice ppl usually mean the instance level inference where n Is assumed to be known/given (n=1 or sample size) and all that had to be learned is the probability vector just like for binomial models it's common to learn "p" , not also n.  Will update comment to clarify and add references  
  
  ---
  
  