Answers.Body: <p>Pasting an answer here from a colleague, credits go to Amelie Groud</p>
  <p>&quot;we encountered the same issue during our PoC, not only with encoding but also with normalization (seeing &quot;age=0.18&quot; wasn't super meaningful).</p>
  <p>If you are using model-agnostic techniques (such as Lime, PDP or ANCHOR), you should be able to apply XAI on your original data, before applying the encoding.
  With these techniques, usually we need 2 main elements: 1. an input dataset and 2. a &quot;predict()&quot; function (calling your trained model).
  From there you have 2 possibilities:</p>
  <ul>
  <li>use your already transformed data and the predict function of your trained model (which seems to be what you are describing)</li>
  <li>or, use the original, unchanged data and have a custom predict function which first apply the transformation needed (encoding, normalization, etc.) and then call the predict() function of your trained model.</li>
  </ul>
  <p>If you choose the 2nd option, then you are free to choose any encoding strategy you see fit. Plus, most XAI techniques have some embedded capacity to deal with categorical data so instead of seeing &quot;pink=1, blue=0, green=0&quot;, you will see &quot;color=pink&quot;.</p>
  <p>Here is an example with LIME but it works similarly for other XAI techniques: <a href="https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html" rel="nofollow noreferrer">https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html</a>
  with the predict function defined as <code>predict_fn = lambda x: rf.predict_proba(encoder.transform(x))</code>&quot;</p>
  <p>Please feel free to add more answers/views if you have a different way of dealing with the question at hand</p>
  <p>Regards,
  Koen</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T07:41:26.527Z
Answers.LastActivityDate: 2021-03-12T07:41:26.527Z
Answers.UserDisplayName: Koen
Answers.Body: <p>Personally I would chose one hot encoding as it is statistically correct/model friendly. Moreover, you can always provide additional help/tools to aid explainability. Lastly even if you add the nth column, you still need some idea about the working of model(and the boundaries it created while training) to interpret the result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T09:11:14.670Z
Answers.LastActivityDate: 2021-03-11T10:32:19.553Z
Answers.LastEditDate: 2021-03-11T10:32:19.553Z
Answers.UserDisplayName: user45344
Body: <p>When creating artificial columns for your categorical variables there are two mainstream methods you could use:</p>
  <p><em>Disclaimer: For this example, I use the following definitions of dummy variables and one-hot-encoding. I'm aware both methods can be used to either return <code>n</code> or <code>n-1</code> columns.</em><br />
  <em>Dummy variables</em>: each category is converted to it's own column and the value 0 or 1 indicates if that category is present for each record<br />
  <em>one-hot-encoding</em>: similar to dummy variables, but one column is dropped, as its value can be derived from the other columns. This is to prevent multicollinearity and the <a href="https://en.wikipedia.org/wiki/Dummy_variable_(statistics)" rel="nofollow noreferrer">dummy variable trap</a>.</p>
  <p>As an arbitrary example, let's take people's favorite color: pink, blue and green. For a person who's favorite color is pink, the dummy and one-hot-encoded data would look as follows:</p>
  <p>dummy variables</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>person_id</th>
  <th>favorite_color_pink</th>
  <th>favorite_color_blue</th>
  <th>favorite_color_green</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>xyz</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>one-hot-encoded variables</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>person_id</th>
  <th>favorite_color_blue</th>
  <th>favorite_color_green</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>xyz</td>
  <td>0</td>
  <td>0</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>From a statistics point of view, I would use the one-hot encoded columns to build my model. In addition, <em>I</em> can infer the favorite color is pink, because I encoded the variables.<br />
  However, when I'm applying XAI to explain the prediction to someone else and <em>they</em> see the favorite color wasn't blue or green. I'm not so sure they will infer the favorite color was pink unless it's explicitly stated. So using dummy variables might serve explainability better, but brings other risks..</p>
  <p>Are there any best practices on this?</p>
  
Tags: <reference-request><explainable-ai>
Title: One hot encoding vs dummy variables best practices for explainable AI (XAI)
UserDisplayName: Koen
ThreadText: # One hot encoding vs dummy variables best practices for explainable AI (XAI)
  
  (Asked by: username_0 on 2021-03-10)
  
  When creating artificial columns for your categorical variables there are two mainstream methods you could use:
  
  *Disclaimer: For this example, I use the following definitions of dummy variables and one-hot-encoding. I'm aware both methods can be used to either return `n` or `n-1` columns.*  
  *Dummy variables*: each category is converted to it's own column and the value 0 or 1 indicates if that category is present for each record  
  *one-hot-encoding*: similar to dummy variables, but one column is dropped, as its value can be derived from the other columns. This is to prevent multicollinearity and the [dummy variable trap](https://en.wikipedia.org/wiki/Dummy_variable_(statistics)).
  
  As an arbitrary example, let's take people's favorite color: pink, blue and green. For a person who's favorite color is pink, the dummy and one-hot-encoded data would look as follows:
  
  dummy variables
  
  | person\_id | favorite\_color\_pink | favorite\_color\_blue | favorite\_color\_green |
  | --- | --- | --- | --- |
  | xyz | 1 | 0 | 0 |
  
  one-hot-encoded variables
  
  | person\_id | favorite\_color\_blue | favorite\_color\_green |
  | --- | --- | --- |
  | xyz | 0 | 0 |
  
  From a statistics point of view, I would use the one-hot encoded columns to build my model. In addition, *I* can infer the favorite color is pink, because I encoded the variables.  
  However, when I'm applying XAI to explain the prediction to someone else and *they* see the favorite color wasn't blue or green. I'm not so sure they will infer the favorite color was pink unless it's explicitly stated. So using dummy variables might serve explainability better, but brings other risks..
  
  Are there any best practices on this?
  
  **Tags:** reference-request, explainable-ai
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2021-03-12. Score: 0 | Accepted answer
  
  Pasting an answer here from a colleague, credits go to Amelie Groud
  
  "we encountered the same issue during our PoC, not only with encoding but also with normalization (seeing "age=0.18" wasn't super meaningful).
  
  If you are using model-agnostic techniques (such as Lime, PDP or ANCHOR), you should be able to apply XAI on your original data, before applying the encoding.
  With these techniques, usually we need 2 main elements: 1. an input dataset and 2. a "predict()" function (calling your trained model).
  From there you have 2 possibilities:
  
  * use your already transformed data and the predict function of your trained model (which seems to be what you are describing)
  * or, use the original, unchanged data and have a custom predict function which first apply the transformation needed (encoding, normalization, etc.) and then call the predict() function of your trained model.
  
  If you choose the 2nd option, then you are free to choose any encoding strategy you see fit. Plus, most XAI techniques have some embedded capacity to deal with categorical data so instead of seeing "pink=1, blue=0, green=0", you will see "color=pink".
  
  Here is an example with LIME but it works similarly for other XAI techniques: <https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html>
  with the predict function defined as `predict_fn = lambda x: rf.predict_proba(encoder.transform(x))`"
  
  Please feel free to add more answers/views if you have a different way of dealing with the question at hand
  
  Regards,
  username_0
  
  
  ---
  
  ### Answer by username_1 on 2021-03-11. Score: 0
  
  Personally I would chose one hot encoding as it is statistically correct/model friendly. Moreover, you can always provide additional help/tools to aid explainability. Lastly even if you add the nth column, you still need some idea about the working of model(and the boundaries it created while training) to interpret the result.
  
  
  ---
  
  