Answers.Body: <ol>
  <li>Distributional mismatch, the value that you pick might be out of distribution, and if inside the distribution, might be that it correspond to the correct one for some samples</li>
  <li>Again, distributional mismatch</li>
  <li>Might be expensive</li>
  </ol>
  <p>Shuffling guarantees you that the noisy value still comes from the same distribution, thus if it comes from the distribution and doesn't create a lot of change then <span class="math-container">$D_{KL}(p(y|x)||p(y))$</span> it's very small, which means that the two distributions are similar and that <span class="math-container">$x$</span> is irrelevant</p>
  <p>As for all ML algorithms, you have tradeoffs, and there is no specific reason why you should use a method over the other, if two are theoretically sound, then it's up to your experience and use case to use one over the other</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:00:10.187Z
Answers.LastActivityDate: 2023-08-06T10:00:10.187Z
Answers.UserDisplayName: Alberto
Body: <p>When calculating permutation importance for a certain feature, that feature is shuffled randomly and predictions with the shuffled feature are compared to predictions with the feature in its original order.</p>
  <p>What are the advantages of this over filling the feature up with a certain value, randomly generated values, deleting the feature and retraining + predicting, and other possible alternatives?</p>
  <p>Note:
  all the alternatives that I mentioned like filling the feature with randomly generated values would be used to predict several values and then those values would be compared with the standard ones</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-07T10:59:14.713Z
Comments.Text: Could you please clarify whether you are asking "Why use permutation importance?", or "Why use random shuffling to measure permutation importance?". Some of your suggested alternatives have nothing to do with measuring anything about permutations, and it is not clear whether that is because you have misunderstood what permutation importance is measuring, or whether you are questioning why someone might be interested in it.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-07T17:54:28.960Z
Comments.Text: @NeilSlater it is likely because I have misunderstood and am attempting to clarify with this question.
Comments.UserDisplayName: catasaurus
Tags: <feature-engineering>
Title: permutation importance shuffling
UserDisplayName: catasaurus
ThreadText: # permutation importance shuffling
  
  (Asked by: username_0 on 2023-08-06)
  
  When calculating permutation importance for a certain feature, that feature is shuffled randomly and predictions with the shuffled feature are compared to predictions with the feature in its original order.
  
  What are the advantages of this over filling the feature up with a certain value, randomly generated values, deleting the feature and retraining + predicting, and other possible alternatives?
  
  Note:
  all the alternatives that I mentioned like filling the feature with randomly generated values would be used to predict several values and then those values would be compared with the standard ones
  
  **Tags:** feature-engineering
  
  > **Comments:**
  > **username_1**: Could you please clarify whether you are asking "Why use permutation importance?", or "Why use random shuffling to measure permutation importance?". Some of your suggested alternatives have nothing to do with measuring anything about permutations, and it is not clear whether that is because you have misunderstood what permutation importance is measuring, or whether you are questioning why someone might be interested in it.  
  > **username_0**: @username_1 it is likely because I have misunderstood and am attempting to clarify with this question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-06. Score: 2 | Accepted answer
  
  1. Distributional mismatch, the value that you pick might be out of distribution, and if inside the distribution, might be that it correspond to the correct one for some samples
  2. Again, distributional mismatch
  3. Might be expensive
  
  Shuffling guarantees you that the noisy value still comes from the same distribution, thus if it comes from the distribution and doesn't create a lot of change then $D\_{KL}(p(y|x)||p(y))$ it's very small, which means that the two distributions are similar and that $x$ is irrelevant
  
  As for all ML algorithms, you have tradeoffs, and there is no specific reason why you should use a method over the other, if two are theoretically sound, then it's up to your experience and use case to use one over the other
  
  
  ---
  
  