Answers.Body: <p>In a simple linear model of the form <span class="math-container">$y = \beta_0 + \beta_1 x $</span> we can see that increasing <span class="math-container">$x$</span> by a unit will increase the prediction on <span class="math-container">$y$</span> by <span class="math-container">$\beta_1$</span>. Here we can completely determine what the effect on the models prediction will be by increasing <span class="math-container">$x$</span>. With more complex models such as neural networks it is much more difficult to tell due to all the calculations that a single data point is involved in. For instance, in a CNN as you mentioned, if I changed the value of a pixel in an image we were passing through the CNN you wouldn't really be able to tell me exactly the effect this would have on the prediction like you can with the linear model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-09T10:14:24.220Z
Answers.Text: so, does that make tree-based model non-interpretable as well? since it wouldn't be completely obvious how change in an input affects the final outputs after having gauged through all the trees in the model?
Answers.UserDisplayName: Naveen Reddy Marthala
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-09T10:19:53.833Z
Answers.Text: @NaveenKumar Decision trees are interpretable to the extent that you know that at every node there's a decision based on an "if/else" statement, so you can recover how a prediction is made by following the path through the tree. In that sense, decision trees are interpretable (i.e. you know that if it was classified as "X", it's because at node $y$ a certain decision was made). However, I would say that interpretability depends on what the human really wants to know.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-09T10:55:42.450Z
Answers.Text: @NaveenKumar nbro pretty much answered your question for me, and I agree it depends on what the human really wants to know. For instance, my background is statistics and I know that medical statisticians would have no interest in using e.g. a Neural Network because you wouldn't be able to tell which features are having the effect on the model; in machine learning often the case is that you don't get how the model works so long as it does, which is why more complex models are popular.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-09T10:03:03.957Z
Answers.LastActivityDate: 2020-10-09T10:03:03.957Z
Answers.UserDisplayName: David
Body: <p>From <a href="https://christophm.github.io/interpretable-ml-book/interpretability-importance.html" rel="nofollow noreferrer">this page in Interpretable-ml book</a> and <a href="https://www.analyticsvidhya.com/blog/2019/08/decoding-black-box-step-by-step-guide-interpretable-machine-learning-models-python/" rel="nofollow noreferrer">this article on Analytics Vidhya</a>, it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion.</p>
  <p>In linear regression, new data will be multiplied with weights and bias will be added to make a prediction.</p>
  <p>And in boosted tree models, it is possible to <a href="https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.plot_tree" rel="nofollow noreferrer">plot all the decisions</a> as trees that results in a prediction.</p>
  <p>And in feed-forward neural networks, we will have weights and biases just like linear regression and we just multiply weights and add bias at each layer, limiting values to some extent using some kind of activation function at every layer, arriving finally at prediction.</p>
  <p>In CNNs, it is possible to see what happens to the input after having passed through a CNN block and what features are extracted after pooling (ref: <a href="https://www.kaggle.com/aakashnain/what-does-a-cnn-see?scriptVersionId=11003133" rel="nofollow noreferrer">what does a CNN see?</a>).</p>
  <p>Like I stated above, one can easily know what happens inside an ML model to make a prediction or conclusion. And I am unclear as to what makes them un-interpretable!. So, what exactly makes an algorithm or it's results un-interpretable or why are these called black box models? Or am I missing something?</p>
  
Tags: <machine-learning><deep-learning><deep-neural-networks><explainable-ai><black-box>
Title: What exactly is an interpretable machine learning model?
UserDisplayName: Naveen Reddy Marthala
ThreadText: # What exactly is an interpretable machine learning model?
  
  (Asked by: username_0 on 2020-10-08)
  
  From [this page in Interpretable-ml book](https://christophm.github.io/interpretable-ml-book/interpretability-importance.html) and [this article on Analytics Vidhya](https://www.analyticsvidhya.com/blog/2019/08/decoding-black-box-step-by-step-guide-interpretable-machine-learning-models-python/), it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion.
  
  In linear regression, new data will be multiplied with weights and bias will be added to make a prediction.
  
  And in boosted tree models, it is possible to [plot all the decisions](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.plot_tree) as trees that results in a prediction.
  
  And in feed-forward neural networks, we will have weights and biases just like linear regression and we just multiply weights and add bias at each layer, limiting values to some extent using some kind of activation function at every layer, arriving finally at prediction.
  
  In CNNs, it is possible to see what happens to the input after having passed through a CNN block and what features are extracted after pooling (ref: [what does a CNN see?](https://www.kaggle.com/aakashnain/what-does-a-cnn-see?scriptVersionId=11003133)).
  
  Like I stated above, one can easily know what happens inside an ML model to make a prediction or conclusion. And I am unclear as to what makes them un-interpretable!. So, what exactly makes an algorithm or it's results un-interpretable or why are these called black box models? Or am I missing something?
  
  **Tags:** machine-learning, deep-learning, deep-neural-networks, explainable-ai, black-box
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-09. Score: 2 | Accepted answer
  
  In a simple linear model of the form $y = \beta\_0 + \beta\_1 x $ we can see that increasing $x$ by a unit will increase the prediction on $y$ by $\beta\_1$. Here we can completely determine what the effect on the models prediction will be by increasing $x$. With more complex models such as neural networks it is much more difficult to tell due to all the calculations that a single data point is involved in. For instance, in a CNN as you mentioned, if I changed the value of a pixel in an image we were passing through the CNN you wouldn't really be able to tell me exactly the effect this would have on the prediction like you can with the linear model.
  
  > **Comments:**
  > **username_0**: so, does that make tree-based model non-interpretable as well? since it wouldn't be completely obvious how change in an input affects the final outputs after having gauged through all the trees in the model?  
  > **username_2**: @NaveenKumar Decision trees are interpretable to the extent that you know that at every node there's a decision based on an "if/else" statement, so you can recover how a prediction is made by following the path through the tree. In that sense, decision trees are interpretable (i.e. you know that if it was classified as "X", it's because at node $y$ a certain decision was made). However, I would say that interpretability depends on what the human really wants to know.  
  > **username_1**: @NaveenKumar nbro pretty much answered your question for me, and I agree it depends on what the human really wants to know. For instance, my background is statistics and I know that medical statisticians would have no interest in using e.g. a Neural Network because you wouldn't be able to tell which features are having the effect on the model; in machine learning often the case is that you don't get how the model works so long as it does, which is why more complex models are popular.  
  
  ---
  
  