Answers.Body: <p>From the two example and the explanation that you provide I wouldn't say that the column text contains sentences, it looks more like gibberish that couldn't fit in any other column, and hence stuck for convenience in a unique generic one.</p>
  <p>I think that here the best way is to perform a bit more data exploration, and try to split that feature into several categorical and numerical features. You mentioned already delivery month and price, those could be extract and become features themselves. And for all the extra stuff that remain I would just create a binary feature similar to &quot;contains_extra_info&quot; yes/no.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-28T16:16:52.917Z
Answers.LastActivityDate: 2022-07-28T16:16:52.917Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual sentence with several pieces of information that outlines prices of something, delivery month, product name and sometimes some other info. That column is basically a text message and its format can vary.</p>
  <p>Example DataFrame looks like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Name</th>
  <th>User</th>
  <th>Text</th>
  <th>Target</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>sarah</td>
  <td>noro</td>
  <td>@-23.50 july 380/crx</td>
  <td>CRX Laptop</td>
  </tr>
  <tr>
  <td>john</td>
  <td>simons</td>
  <td>(atlas5) sep nc8 npc 131.5 @ 132.5</td>
  <td>NPC Playstation</td>
  </tr>
  <tr>
  <td>...</td>
  <td>...</td>
  <td>...</td>
  <td>...</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>The columns <em><strong>Name, User, Text</strong></em> are features, and column <em><strong>Target</strong></em> is the target column that I want to predict. I would like to use a <strong>classifier</strong> (Random Forest, or Neural Net, or GBDT) to classify the Target based on other three columns.</p>
  <p>I can category encode the columns <em>Name, User, Target</em> as I know how many of unique names, users, and targets there are.</p>
  <ol>
  <li>How do I encode the <strong>Text</strong> column as it is a sentence?</li>
  <li>Would it be better to regex split the <strong>Text</strong> column and simplify it (remove numbers etc) or just feed in the whole thing?</li>
  </ol>
  
Tags: <natural-language-processing><classification><data-preprocessing>
Title: How to encode both sentences and categorical data?
UserDisplayName: MilTom
ThreadText: # How to encode both sentences and categorical data?
  
  (Asked by: username_0 on 2022-07-28)
  
  I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual sentence with several pieces of information that outlines prices of something, delivery month, product name and sometimes some other info. That column is basically a text message and its format can vary.
  
  Example DataFrame looks like this:
  
  | Name | User | Text | Target |
  | --- | --- | --- | --- |
  | sarah | noro | @-23.50 july 380/crx | CRX Laptop |
  | john | simons | (atlas5) sep nc8 npc 131.5 @ 132.5 | NPC Playstation |
  | ... | ... | ... | ... |
  
  The columns ***Name, User, Text*** are features, and column ***Target*** is the target column that I want to predict. I would like to use a **classifier** (Random Forest, or Neural Net, or GBDT) to classify the Target based on other three columns.
  
  I can category encode the columns *Name, User, Target* as I know how many of unique names, users, and targets there are.
  
  1. How do I encode the **Text** column as it is a sentence?
  2. Would it be better to regex split the **Text** column and simplify it (remove numbers etc) or just feed in the whole thing?
  
  **Tags:** natural-language-processing, classification, data-preprocessing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-28. Score: 2
  
  From the two example and the explanation that you provide I wouldn't say that the column text contains sentences, it looks more like gibberish that couldn't fit in any other column, and hence stuck for convenience in a unique generic one.
  
  I think that here the best way is to perform a bit more data exploration, and try to split that feature into several categorical and numerical features. You mentioned already delivery month and price, those could be extract and become features themselves. And for all the extra stuff that remain I would just create a binary feature similar to "contains\_extra\_info" yes/no.
  
  
  ---
  
  