Answers.Body: <p><a href="https://arxiv.org/pdf/1901.04592.pdf" rel="nofollow noreferrer">There is not a widely accepted definition of explainable AI (XAI)</a>. However, as a rule of thumb (my rule of thumb), if you can't explain it <em>easily</em> to a layperson (or even an expert), then the model or algorithm is not (very) interpretable. There are other concepts related to XAI, such as accountability (who is responsible for what?), transparency and fairness.</p>
  
  <p>For example, the final decision of (trained) decision tree can easily be explained to (almost) any person, so a (trained) decision tree is a relatively interpretable model. See the chapter <a href="https://christophm.github.io/interpretable-ml-book/tree.html" rel="nofollow noreferrer">4.4. Decision Tree</a> of the book <a href="https://christophm.github.io/interpretable-ml-book/" rel="nofollow noreferrer">Interpretable Machine Learning: A Guide for Making Black Box Models Explainable</a>.</p>
  
  <p>An artificial neural network (ANN) is usually considered not very interpretable because, unless you attempt to understand which parts of the network contribute to the output of the ANN (for example, with the technique <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140" rel="nofollow noreferrer">layer-wise relevance propagation</a>), then you cannot immediately or easily understand the output or decision of the ANN, given that an ANN involves many non-linear functions, which produce unintuitive outcomes. In other words, it is more difficult to attribute the contributions of each unit of an ANN to the output of the same ANN than to explain e.g. the decision of a decision tree. </p>
  
  <p>In the context of deep reinforcement learning (DRL), the ANN is used to approximate the value or policy functions. This approximation is, in the first place, the main reason behind the low interpretability of deep RL models.</p>
  
  <p>Q-learning is an algorithm, so it is not a model, like an ANN. Q-learning is used to learn a state-action value function, denoted with <span class="math-container">$Q: S \times A \rightarrow \mathbb{R}$</span>, which can then be used to derive another function, <a href="https://ai.stackexchange.com/q/12274/2444">the policy</a>, which can then be used to take actions. In a way, Q-learning is similar to gradient descent, because both are machine learning (or optimization) algorithms. The <span class="math-container">$Q$</span> function is a model of the environment, given that, for each state, it represents the expected amount of reward that can be obtained, so, in a certain way, the learned <span class="math-container">$Q$</span> function represents a prediction of reward. </p>
  
  <p>Is the learned tabular <span class="math-container">$Q$</span> function interpretable? Yes, it is <em>relatively</em> interpretable, but how much? What kind of interpretation do you really need? It depends on the context and people that need the interpretation or explanation. A reinforcement learning researcher will usually be satisfied with the usual explanation of the inner workings of <span class="math-container">$Q$</span>-learning, Markov decision processes, etc., because the usual RL researcher is not concerned with the really important problems that involve the life of people and other beings. However, for example, in the context of healthcare, doctors might not just be interested in the explanation "expected maximum future reward", but they might also be interested in the environment, <a href="https://ai.stackexchange.com/q/12908/2444">the credit assignment problem</a>, the meaning and effectiveness of the reward function with respect to the actual problem that needs to be solved, in a probabilistic interpretation of the results (rather than just a mere action that needs to be taken), possible alternative good actions, etc.</p>
  
  <p>Recently, there have been some attempts to make RL and, in particular, deep RL more interpretable and explainable. In the paper <a href="https://arxiv.org/pdf/1804.02477.pdf" rel="nofollow noreferrer">Programmatically Interpretable Reinforcement Learning</a> (2019), Verma et al. propose a more interpretable (than deep RL) RL framework that is based on the idea of learning policies that are represented in a human-readable language. In the paper <a href="https://arxiv.org/pdf/1905.10404.pdf" rel="nofollow noreferrer">InfoRL: Interpretable Reinforcement Learning using Information Maximization</a> (2019), the authors focus on learning multiple ways of solving the same task and they claim that their approach provides more interpretability. In the paper <a href="http://www.ecmlpkdd2018.org/wp-content/uploads/2018/09/246.pdf" rel="nofollow noreferrer">Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees</a> (2018), the authors also claim that their approach <em>facilitates understanding the network's learned knowledge by analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs</em>.</p>
  
  <p>To conclude, deep RL should not necessarily be avoided: it depends on the context (e.g., it is usually perfectly fine to use deep RL to solve video games). However, in cases where liability is an issue, then deep RL should also be explainable or more explainable alternatives should also be taken into account.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-12T21:58:44.407Z
Answers.LastActivityDate: 2019-08-13T21:15:30.703Z
Answers.LastEditDate: 2019-08-13T21:15:30.703Z
Answers.UserDisplayName: nbro
Body: <p>I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent structure to it, and should not require resorting to deep Q-learning. Based on my hypothesis, I managed to create a tabular Q-learning based algorithm which uses limited domain knowledge to perform on-par/outperform the deep Q-learning based approaches.</p>
  
  <p>Given that model <em>interpretability</em> is a subjective and sometimes vague topic, I was wondering if my algorithm should be considered interpretable. The way I understand it, the lack of interpretability in deep-learning-based models stems from the stochastic gradient descent step. However, in case of tabular Q-learning, every chosen action can always be traced back to a finite set of action-value pairs, which in turn are a deterministic function of inputs of the algorithm, although over multiple training episodes.</p>
  
  <p>I believe in using deep-learning-based approaches conservatively only when absolutely required. However, I am not sure how to justify this in my paper without wading into the debated topic of model interpretability. I would greatly appreciate any suggestions/opinions regarding this. </p>
  
Tags: <neural-networks><deep-learning><reinforcement-learning><q-learning><explainable-ai>
Title: Is tabular Q-learning considered interpretable?
UserDisplayName: harshal.c
ThreadText: # Is tabular Q-learning considered interpretable?
  
  (Asked by: username_0 on 2019-08-12)
  
  I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent structure to it, and should not require resorting to deep Q-learning. Based on my hypothesis, I managed to create a tabular Q-learning based algorithm which uses limited domain knowledge to perform on-par/outperform the deep Q-learning based approaches.
  
  Given that model *interpretability* is a subjective and sometimes vague topic, I was wondering if my algorithm should be considered interpretable. The way I understand it, the lack of interpretability in deep-learning-based models stems from the stochastic gradient descent step. However, in case of tabular Q-learning, every chosen action can always be traced back to a finite set of action-value pairs, which in turn are a deterministic function of inputs of the algorithm, although over multiple training episodes.
  
  I believe in using deep-learning-based approaches conservatively only when absolutely required. However, I am not sure how to justify this in my paper without wading into the debated topic of model interpretability. I would greatly appreciate any suggestions/opinions regarding this.
  
  **Tags:** neural-networks, deep-learning, reinforcement-learning, q-learning, explainable-ai
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-12. Score: 2
  
  [There is not a widely accepted definition of explainable AI (XAI)](https://arxiv.org/pdf/1901.04592.pdf). However, as a rule of thumb (my rule of thumb), if you can't explain it *easily* to a layperson (or even an expert), then the model or algorithm is not (very) interpretable. There are other concepts related to XAI, such as accountability (who is responsible for what?), transparency and fairness.
  
  For example, the final decision of (trained) decision tree can easily be explained to (almost) any person, so a (trained) decision tree is a relatively interpretable model. See the chapter [4.4. Decision Tree](https://christophm.github.io/interpretable-ml-book/tree.html) of the book [Interpretable Machine Learning: A Guide for Making Black Box Models Explainable](https://christophm.github.io/interpretable-ml-book/).
  
  An artificial neural network (ANN) is usually considered not very interpretable because, unless you attempt to understand which parts of the network contribute to the output of the ANN (for example, with the technique [layer-wise relevance propagation](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140)), then you cannot immediately or easily understand the output or decision of the ANN, given that an ANN involves many non-linear functions, which produce unintuitive outcomes. In other words, it is more difficult to attribute the contributions of each unit of an ANN to the output of the same ANN than to explain e.g. the decision of a decision tree.
  
  In the context of deep reinforcement learning (DRL), the ANN is used to approximate the value or policy functions. This approximation is, in the first place, the main reason behind the low interpretability of deep RL models.
  
  Q-learning is an algorithm, so it is not a model, like an ANN. Q-learning is used to learn a state-action value function, denoted with $Q: S \times A \rightarrow \mathbb{R}$, which can then be used to derive another function, [the policy](https://ai.stackexchange.com/q/12274/2444), which can then be used to take actions. In a way, Q-learning is similar to gradient descent, because both are machine learning (or optimization) algorithms. The $Q$ function is a model of the environment, given that, for each state, it represents the expected amount of reward that can be obtained, so, in a certain way, the learned $Q$ function represents a prediction of reward.
  
  Is the learned tabular $Q$ function interpretable? Yes, it is *relatively* interpretable, but how much? What kind of interpretation do you really need? It depends on the context and people that need the interpretation or explanation. A reinforcement learning researcher will usually be satisfied with the usual explanation of the inner workings of $Q$-learning, Markov decision processes, etc., because the usual RL researcher is not concerned with the really important problems that involve the life of people and other beings. However, for example, in the context of healthcare, doctors might not just be interested in the explanation "expected maximum future reward", but they might also be interested in the environment, [the credit assignment problem](https://ai.stackexchange.com/q/12908/2444), the meaning and effectiveness of the reward function with respect to the actual problem that needs to be solved, in a probabilistic interpretation of the results (rather than just a mere action that needs to be taken), possible alternative good actions, etc.
  
  Recently, there have been some attempts to make RL and, in particular, deep RL more interpretable and explainable. In the paper [Programmatically Interpretable Reinforcement Learning](https://arxiv.org/pdf/1804.02477.pdf) (2019), Verma et al. propose a more interpretable (than deep RL) RL framework that is based on the idea of learning policies that are represented in a human-readable language. In the paper [InfoRL: Interpretable Reinforcement Learning using Information Maximization](https://arxiv.org/pdf/1905.10404.pdf) (2019), the authors focus on learning multiple ways of solving the same task and they claim that their approach provides more interpretability. In the paper [Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees](http://www.ecmlpkdd2018.org/wp-content/uploads/2018/09/246.pdf) (2018), the authors also claim that their approach *facilitates understanding the network's learned knowledge by analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs*.
  
  To conclude, deep RL should not necessarily be avoided: it depends on the context (e.g., it is usually perfectly fine to use deep RL to solve video games). However, in cases where liability is an issue, then deep RL should also be explainable or more explainable alternatives should also be taken into account.
  
  
  ---
  
  