Answers.Body: <p><em>Explainable AI</em> and <em>model interpretability</em> are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (often tremendous) success of deep learning models in  various tasks, plus the necessity of algorithmic fairness &amp; accountability.</p>
  <p>Here are some state of the art algorithms and approaches, together with implementations and frameworks.</p>
  <hr />
  <p><strong>Model-agnostic approaches</strong></p>
  <ul>
  <li><p>LIME: Local Interpretable Model-agnostic Explanations (<a href="https://arxiv.org/abs/1602.04938" rel="noreferrer">paper</a>, <a href="https://github.com/marcotcr/lime" rel="noreferrer">code</a>, <a href="https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime" rel="noreferrer">blog post</a>, <a href="https://cran.r-project.org/web/packages/lime/index.html" rel="noreferrer">R port</a>)</p>
  </li>
  <li><p>SHAP: A Unified Approach to Interpreting Model Predictions (<a href="https://arxiv.org/abs/1705.07874" rel="noreferrer">paper</a>, <a href="https://github.com/slundberg/shap" rel="noreferrer">Python package</a>, <a href="https://github.com/redichh/ShapleyR" rel="noreferrer">R package</a>). GPU implementation for tree models by NVIDIA using RAPIDS - GPUTreeShap (<a href="https://arxiv.org/abs/2010.13972" rel="noreferrer">paper</a>, <a href="https://github.com/rapidsai/gputreeshap" rel="noreferrer">code</a>, <a href="https://developer.nvidia.com/blog/explaining-and-accelerating-machine-learning-for-loan-delinquencies/" rel="noreferrer">blog post</a>)</p>
  </li>
  <li><p>Anchors: High-Precision Model-Agnostic Explanations (<a href="https://homes.cs.washington.edu/%7Emarcotcr/aaai18.pdf" rel="noreferrer">paper</a>, <a href="https://github.com/marcotcr/anchor" rel="noreferrer">authors' Python code</a>, <a href="https://github.com/viadee/javaAnchorExplainer" rel="noreferrer">Java</a> implementation)</p>
  </li>
  <li><p>Diverse Counterfactual Explanations (DiCE) by Microsoft (<a href="https://arxiv.org/abs/1905.07697" rel="noreferrer">paper</a>, <a href="https://github.com/microsoft/dice" rel="noreferrer">code</a>, <a href="https://www.microsoft.com/en-us/research/blog/open-source-library-provides-explanation-for-machine-learning-through-diverse-counterfactuals/" rel="noreferrer">blog post</a>)</p>
  </li>
  <li><p><a href="https://arxiv.org/abs/1602.07043" rel="noreferrer">Black Box Auditing</a> and <a href="https://arxiv.org/abs/1412.3756" rel="noreferrer">Certifying and Removing Disparate Impact</a> (authors' <a href="https://github.com/algofairness/BlackBoxAuditing" rel="noreferrer">Python code</a>)</p>
  </li>
  <li><p>FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (<a href="http://blog.fastforwardlabs.com/2017/03/09/fairml-auditing-black-box-predictive-models.html" rel="noreferrer">blog post</a>, <a href="https://arxiv.org/abs/1611.04967" rel="noreferrer">paper</a>, <a href="https://github.com/adebayoj/fairml" rel="noreferrer">code</a>)</p>
  </li>
  </ul>
  <p>SHAP seems to enjoy <a href="https://twitter.com/christophmolnar/status/1162460768631177216?s=11" rel="noreferrer">high popularity</a> among practitioners; the method has firm theoretical foundations on co-operational game theory (Shapley values), and it has in a great degree integrated the LIME approach under a common framework. Although model-agnostic, specific &amp; efficient implementations are available for neural networks (<a href="https://shap.readthedocs.io/en/latest/generated/shap.DeepExplainer.html" rel="noreferrer">DeepExplainer</a>) and tree ensembles (<a href="https://shap.readthedocs.io/en/latest/generated/shap.TreeExplainer.html" rel="noreferrer">TreeExplainer</a>, <a href="https://www.nature.com/articles/s42256-019-0138-9.epdf?shared_access_token=RCYPTVkiECUmc0CccSMgXtRgN0jAjWel9jnR3ZoTv0O81kV8DqPb2VXSseRmof0Pl8YSOZy4FHz5vMc3xsxcX6uT10EzEoWo7B-nZQAHJJvBYhQJTT1LnJmpsa48nlgUWrMkThFrEIvZstjQ7Xdc5g%3D%3D" rel="noreferrer">paper</a>).</p>
  <hr />
  <p><strong>Neural network approaches</strong> (mostly, but not exclusively, for computer vision models)</p>
  <ul>
  <li><p>The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (<a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140" rel="noreferrer">2015 paper @ PLoS ONE</a>, <a href="http://www.jmlr.org/papers/v17/15-618.html" rel="noreferrer">2016 paper @ JMLR</a>, <a href="http://heatmapping.org/" rel="noreferrer">project page</a>, <a href="https://github.com/sebastian-lapuschkin/lrp_toolbox" rel="noreferrer">code</a>, <a href="https://github.com/VigneshSrinivasan10/interprettensor" rel="noreferrer">TF Slim wrapper</a>)</p>
  </li>
  <li><p>Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (<a href="https://arxiv.org/abs/1610.02391" rel="noreferrer">paper</a>, authors' <a href="https://github.com/ramprs/grad-cam" rel="noreferrer">Torch code</a>, <a href="https://github.com/Ankush96/grad-cam.tensorflow" rel="noreferrer">Tensorflow code</a>, <a href="https://github.com/meliketoy/gradcam.pytorch" rel="noreferrer">PyTorch code</a>, yet another <a href="https://github.com/jacobgil/pytorch-grad-cam" rel="noreferrer">Pytorch implementation</a>, Keras <a href="http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb" rel="noreferrer">example notebook</a>, Coursera <a href="https://www.coursera.org/projects/scene-classification-gradcam" rel="noreferrer">Guided Project</a>)</p>
  </li>
  <li><p>Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (<a href="https://arxiv.org/abs/2008.02312" rel="noreferrer">paper</a>, <a href="https://github.com/Fu0511/XGrad-CAM" rel="noreferrer">code</a>)</p>
  </li>
  <li><p>SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (<a href="https://arxiv.org/abs/1706.05806" rel="noreferrer">paper</a>, <a href="https://github.com/google/svcca" rel="noreferrer">code</a>, <a href="https://ai.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html" rel="noreferrer">Google blog post</a>)</p>
  </li>
  <li><p>TCAV: Testing with Concept Activation Vectors (<a href="https://arxiv.org/abs/1711.11279" rel="noreferrer">ICML 2018 paper</a>, <a href="https://github.com/tensorflow/tcav" rel="noreferrer">Tensorflow code</a>)</p>
  </li>
  <li><p>Integrated Gradients (<a href="https://arxiv.org/abs/1703.01365" rel="noreferrer">paper</a>, <a href="https://github.com/ankurtaly/Integrated-Gradients" rel="noreferrer">code</a>, <a href="https://www.tensorflow.org/tutorials/interpretability/integrated_gradients" rel="noreferrer">Tensorflow tutorial</a>, <a href="https://paperswithcode.com/paper/axiomatic-attribution-for-deep-networks#code" rel="noreferrer">independent implementations</a>)</p>
  </li>
  <li><p>Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (<a href="http://netdissect.csail.mit.edu/" rel="noreferrer">project page</a>, <a href="https://github.com/CSAILVision/NetDissect" rel="noreferrer">Caffe code</a>, <a href="https://github.com/CSAILVision/NetDissect-Lite" rel="noreferrer">PyTorch port</a>)</p>
  </li>
  <li><p>GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (<a href="https://gandissect.csail.mit.edu/" rel="noreferrer">project page</a>, with links to paper &amp; code)</p>
  </li>
  <li><p>Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (<a href="https://arxiv.org/abs/1811.08011" rel="noreferrer">paper</a>, <a href="https://github.com/gudovskiy/e2x" rel="noreferrer">code</a>)</p>
  </li>
  <li><p>Transparecy-by-Design (TbD) networks (<a href="https://arxiv.org/abs/1803.05268" rel="noreferrer">paper</a>, <a href="https://github.com/davidmascharka/tbd-nets" rel="noreferrer">code</a>, <a href="https://mybinder.org/v2/gh/davidmascharka/tbd-nets/binder?filepath=full-vqa-example.ipynb" rel="noreferrer">demo</a>)</p>
  </li>
  <li><p><a href="https://arxiv.org/abs/1711.09784" rel="noreferrer">Distilling a Neural Network Into a Soft Decision Tree</a>, a 2017 paper by Geoff Hinton, with various independent <a href="https://paperswithcode.com/paper/distilling-a-neural-network-into-a-soft" rel="noreferrer">PyTorch implementations</a></p>
  </li>
  <li><p>Understanding Deep Networks via Extremal Perturbations and Smooth Masks (<a href="https://arxiv.org/abs/1910.08485" rel="noreferrer">paper</a>), implemented in <a href="https://github.com/facebookresearch/torchray" rel="noreferrer">TorchRay</a> (see below)</p>
  </li>
  <li><p>Understanding the Role of Individual Units in a Deep Neural Network (<a href="https://arxiv.org/abs/2009.05041" rel="noreferrer">preprint</a>, <a href="https://www.pnas.org/content/early/2020/08/31/1907375117" rel="noreferrer">2020 paper @ PNAS</a>, <a href="https://github.com/davidbau/dissect" rel="noreferrer">code</a>, <a href="https://dissect.csail.mit.edu/" rel="noreferrer">project page</a>)</p>
  </li>
  <li><p>GNNExplainer: Generating Explanations for Graph Neural Networks (<a href="https://proceedings.neurips.cc/paper/2019/file/d80b7040b773199015de6d3b4293c8ff-Paper.pdf" rel="noreferrer">paper</a>, <a href="https://github.com/RexYing/gnn-model-explainer" rel="noreferrer">code</a>)</p>
  </li>
  <li><p>Benchmarking Deep Learning Interpretability in Time Series Predictions (<a href="https://proceedings.neurips.cc//paper/2020/file/47a3893cc405396a5c30d91320572d6d-Paper.pdf" rel="noreferrer">paper</a> @ NeurIPS 2020, <a href="https://github.com/ayaabdelsalam91/TS-Interpretability-Benchmark" rel="noreferrer">code</a> utilizing <a href="https://captum.ai" rel="noreferrer">Captum</a>)</p>
  </li>
  <li><p>Concept Whitening for Interpretable Image Recognition (<a href="https://www.nature.com/articles/s42256-020-00265-z" rel="noreferrer">paper</a>, <a href="https://arxiv.org/abs/2002.01650" rel="noreferrer">preprint</a>, <a href="https://github.com/zhiCHEN96/ConceptWhitening" rel="noreferrer">code</a>)</p>
  </li>
  </ul>
  <hr />
  <p><strong>Libraries &amp; frameworks</strong></p>
  <p>As interpretability moves toward the mainstream, there are already frameworks and toolboxes that incorporate more than one of the algorithms and techniques mentioned and linked above; here is a partial list:</p>
  <ul>
  <li><p>The ELI5 Python library (<a href="https://github.com/TeamHG-Memex/eli5" rel="noreferrer">code</a>, <a href="https://eli5.readthedocs.io/en/latest/" rel="noreferrer">documentation</a>)</p>
  </li>
  <li><p>DALEX - moDel Agnostic Language for Exploration and eXplanation (<a href="https://modeloriented.github.io/DALEX/" rel="noreferrer">homepage</a>, <a href="https://github.com/ModelOriented/DALEX" rel="noreferrer">code</a>, <a href="https://www.jmlr.org/papers/volume19/18-416/18-416.pdf" rel="noreferrer">JMLR paper</a>), part of the <a href="https://modeloriented.github.io/DrWhy/" rel="noreferrer">DrWhy.AI</a> project</p>
  </li>
  <li><p>The What-If tool by Google, a feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (<a href="https://pair-code.github.io/what-if-tool/" rel="noreferrer">project page</a>, <a href="https://github.com/pair-code/what-if-tool" rel="noreferrer">code</a>, <a href="https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html" rel="noreferrer">blog post</a>)</p>
  </li>
  <li><p>The Language Interpretability Tool (LIT) by Google, a visual, interactive model-understanding tool for NLP models (<a href="https://pair-code.github.io/lit/" rel="noreferrer">project page</a>, <a href="https://github.com/PAIR-code/lit/" rel="noreferrer">code</a>, <a href="https://ai.googleblog.com/2020/11/the-language-interpretability-tool-lit.html" rel="noreferrer">blog post</a>)</p>
  </li>
  <li><p>Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (<a href="https://github.com/tensorflow/lucid" rel="noreferrer">code</a>; papers: <a href="https://distill.pub/2017/feature-visualization/" rel="noreferrer">Feature Visualization</a>, <a href="https://distill.pub/2018/building-blocks/" rel="noreferrer">The Building Blocks of Interpretability</a>)</p>
  </li>
  <li><p><a href="https://github.com/facebookresearch/torchray" rel="noreferrer">TorchRay</a> by Facebook, a PyTorch package implementing several visualization methods for deep CNNs</p>
  </li>
  <li><p>iNNvestigate Neural Networks (<a href="https://github.com/albermax/innvestigate" rel="noreferrer">code</a>, <a href="https://jmlr.org/papers/v20/18-540.html" rel="noreferrer">JMLR paper</a>)</p>
  </li>
  <li><p>tf-explain - interpretability methods as Tensorflow 2.0 callbacks (<a href="https://github.com/sicara/tf-explain" rel="noreferrer">code</a>, <a href="https://tf-explain.readthedocs.io/en/latest/" rel="noreferrer">docs</a>, <a href="https://blog.sicara.com/tf-explain-interpretability-tensorflow-2-9438b5846e35" rel="noreferrer">blog post</a>)</p>
  </li>
  <li><p>InterpretML by Microsoft (<a href="https://interpret.ml/" rel="noreferrer">homepage</a>, <a href="https://github.com/Microsoft/interpret" rel="noreferrer">code</a> still in alpha, <a href="https://arxiv.org/abs/1909.09223" rel="noreferrer">paper</a>)</p>
  </li>
  <li><p>Captum by Facebook AI - model interpetability for Pytorch (<a href="https://captum.ai" rel="noreferrer">homepage</a>, <a href="https://github.com/pytorch/captum" rel="noreferrer">code</a>, <a href="https://ai.facebook.com/blog/open-sourcing-captum-a-model-interpretability-library-for-pytorch/" rel="noreferrer">intro blog post</a>)</p>
  </li>
  <li><p>Skater, by Oracle (<a href="https://github.com/oracle/Skater" rel="noreferrer">code</a>, <a href="https://oracle.github.io/Skater/" rel="noreferrer">docs</a>)</p>
  </li>
  <li><p>Alibi, by SeldonIO (<a href="https://github.com/SeldonIO/alibi" rel="noreferrer">code</a>, <a href="https://docs.seldon.io/projects/alibi/en/stable/" rel="noreferrer">docs</a>)</p>
  </li>
  <li><p>AI Explainability 360, commenced by IBM and moved to the Linux Foundation (<a href="https://ai-explainability-360.org/" rel="noreferrer">homepage</a>, <a href="https://github.com/Trusted-AI/AIX360" rel="noreferrer">code</a>, <a href="https://aix360.readthedocs.io/en/latest/" rel="noreferrer">docs</a>, <a href="http://aix360.mybluemix.net/" rel="noreferrer">IBM Bluemix</a>, <a href="https://www.ibm.com/blogs/research/2019/08/ai-explainability-360/" rel="noreferrer">blog post</a>)</p>
  </li>
  <li><p>Ecco: explaining transformer-based NLP models using interactive visualizations (<a href="https://www.eccox.io/" rel="noreferrer">homepage</a>, <a href="https://github.com/jalammar/ecco" rel="noreferrer">code</a>, <a href="https://jalammar.github.io/explaining-transformers/" rel="noreferrer">article</a>).</p>
  </li>
  <li><p>Recipes for Machine Learning Interpretability in H2O Driverless AI (<a href="https://github.com/h2oai/driverlessai-recipes/tree/master/explainers" rel="noreferrer">repo</a>)</p>
  </li>
  </ul>
  <hr />
  <p><strong>Reviews &amp; general papers</strong></p>
  <ul>
  <li><p><a href="https://dl.acm.org/doi/10.1145/3236009" rel="noreferrer">A Survey of Methods for Explaining Black Box Models</a> (2018, ACM Computing Surveys)</p>
  </li>
  <li><p><a href="https://www.pnas.org/content/116/44/22071" rel="noreferrer">Definitions, methods, and applications in interpretable machine learning</a> (2019, PNAS)</p>
  </li>
  <li><p><a href="https://www.nature.com/articles/s42256-019-0048-x" rel="noreferrer">Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead</a> (2019, Nature Machine Intelligence, <a href="https://arxiv.org/abs/1811.10154" rel="noreferrer">preprint</a>)</p>
  </li>
  <li><p><a href="https://www.mdpi.com/2079-9292/8/8/832" rel="noreferrer">Machine Learning Interpretability: A Survey on Methods and Metrics</a> (2019, Electronics)</p>
  </li>
  <li><p><a href="https://arxiv.org/abs/2009.11698" rel="noreferrer">Principles and Practice of Explainable Machine Learning</a> (2020, preprint)</p>
  </li>
  <li><p><a href="https://arxiv.org/abs/2010.09337" rel="noreferrer">Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges</a> (keynote at <a href="https://kdd.isti.cnr.it/xkdd2020/" rel="noreferrer">2020 ECML XKDD workshop</a> by Christoph Molnar, <a href="https://slideslive.com/38933066/interpretable-machine-learning-state-of-the-art-and-challenges" rel="noreferrer">video &amp; slides</a>)</p>
  </li>
  <li><p><a href="https://www.sciencedirect.com/science/article/pii/S1566253519308103" rel="noreferrer">Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI</a> (2020, Information Fusion)</p>
  </li>
  <li><p><a href="https://arxiv.org/abs/2010.10596" rel="noreferrer">Counterfactual Explanations for Machine Learning: A Review</a> (2020, preprint, <a href="https://twitter.com/yudapearl/status/1321038352355790848?s=11" rel="noreferrer">critique</a> by Judea Pearl)</p>
  </li>
  <li><p><a href="https://ff06-2020.fastforwardlabs.com/" rel="noreferrer">Interpretability 2020</a>, an applied research report by Cloudera Fast Forward, updated regularly</p>
  </li>
  <li><p><a href="https://github.com/Eric-Wallace/interpretability-tutorial-emnlp2020" rel="noreferrer">Interpreting Predictions of NLP Models</a> (EMNLP 2020 tutorial)</p>
  </li>
  <li><p>Explainable NLP Datasets (<a href="https://exnlpdatasets.github.io/" rel="noreferrer">site</a>, <a href="https://arxiv.org/abs/2102.12060" rel="noreferrer">preprint</a>, <a href="https://twitter.com/sarahwiegreffe/status/1365057786250489858" rel="noreferrer">highlights</a>)</p>
  </li>
  <li><p><a href="https://arxiv.org/abs/2103.11251" rel="noreferrer">Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges</a></p>
  </li>
  </ul>
  <hr />
  <p><strong>eBooks</strong> (available online)</p>
  <ul>
  <li><p><a href="https://christophm.github.io/interpretable-ml-book/" rel="noreferrer">Interpretable Machine Learning</a>, by Christoph Molnar, with <a href="https://github.com/christophM/iml" rel="noreferrer">R code</a> available</p>
  </li>
  <li><p><a href="https://pbiecek.github.io/ema/" rel="noreferrer">Explanatory Model Analysis</a>, by DALEX creators Przemyslaw Biecek and Tomasz Burzykowski, with both R &amp; Python code snippets</p>
  </li>
  <li><p><a href="https://www.h2o.ai/wp-content/uploads/2019/08/An-Introduction-to-Machine-Learning-Interpretability-Second-Edition.pdf" rel="noreferrer">An Introduction to Machine Learning Interpretability</a> (2nd ed. 2019), by H2O</p>
  </li>
  </ul>
  <hr />
  <p><strong>Online courses &amp; tutorials</strong></p>
  <ul>
  <li><p><a href="https://www.kaggle.com/learn/machine-learning-explainability" rel="noreferrer">Machine Learning Explainability</a>, Kaggle tutorial</p>
  </li>
  <li><p><a href="https://www.coursera.org/projects/scene-classification-gradcam" rel="noreferrer">Explainable AI: Scene Classification and GradCam Visualization</a>, Coursera guided project</p>
  </li>
  <li><p><a href="http://Explainable%20Machine%20Learning%20with%20LIME%20and%20H2O%20in%20R" rel="noreferrer">Explainable Machine Learning with LIME and H2O in R</a>, Coursera guided project</p>
  </li>
  <li><p><a href="https://interpretable-ml-class.github.io/" rel="noreferrer">Interpretability and Explainability in Machine Learning</a>, Harvard COMPSCI 282BR</p>
  </li>
  </ul>
  <hr />
  <p><strong>Other resources</strong></p>
  <ul>
  <li><p><a href="https://explained.ai/" rel="noreferrer">explained.ai</a> blog</p>
  </li>
  <li><p>A <a href="https://twitter.com/ledell/status/995930308947140608" rel="noreferrer">Twitter thread</a>, linking to several interpretation tools available for R</p>
  </li>
  <li><p>A whole bunch of resources in the <a href="https://github.com/jphall663/awesome-machine-learning-interpretability" rel="noreferrer">Awesome Machine Learning Interpetability</a> repo</p>
  </li>
  <li><p>The online comic book (!) <a href="https://betaandbit.github.io/RML/#p=1" rel="noreferrer"><em>The Hitchhiker's Guide to Responsible Machine Learning</em></a>, by the team behind the textbook <em>Explanatory Model Analysis</em> and the DALEX package mentioned above (<a href="https://medium.com/responsibleml/the-hitchhikers-guide-to-responsible-machine-learning-d9179a047319" rel="noreferrer">blog post</a> and <a href="https://medium.com/responsibleml/backstage-the-hitchhikers-guide-to-responsible-machine-learning-3a8735a0f8bb" rel="noreferrer">backstage</a>)</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-02T16:40:53.900Z
Answers.Text: Hey.  According to the paper, GNNEXPLAINER may be model-agnostic rather than neural-network-based. I paraphrase from the paper "Here we propose GNNEXPLAINER, the first general, *model-agnostic* approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task."
Answers.UserDisplayName: Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T21:55:15.390Z
Answers.LastActivityDate: 2021-12-03T15:23:49.830Z
Answers.LastEditDate: 2021-12-03T15:23:49.830Z
Answers.UserDisplayName: desertnaut
Answers.Body: <p>There are a few <a href="https://christophm.github.io/interpretable-ml-book/agnostic.html" rel="nofollow noreferrer">XAI techniques</a> that are (partially) agnostic to the model to be interpreted</p>
  
  <ul>
  <li><p>Layer-wise relevance propagation (LRP), introduced in <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140" rel="nofollow noreferrer">On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation</a> (2015)</p></li>
  <li><p>Local Interpretable Model-agnostic Explanations (LIME), introduced in <a href="https://arxiv.org/pdf/1602.04938v1.pdf" rel="nofollow noreferrer">"Why Should I Trust You?" Explaining the Predictions of Any Classifier</a> (2016)</p></li>
  <li><p>Model Agnostic Contrastive Explanations Method (MACEM), introduced in <a href="https://arxiv.org/pdf/1906.00117.pdf" rel="nofollow noreferrer">Model Agnostic Contrastive Explanations for Structured Data</a> (2019)</p></li>
  </ul>
  
  <p>There are also ML models that are not considered black boxes and that are thus <a href="https://christophm.github.io/interpretable-ml-book/simple.html" rel="nofollow noreferrer">more interpretable than black boxes</a>, such as</p>
  
  <ul>
  <li>linear models (e.g. linear regression)</li>
  <li>decision trees</li>
  <li>naive Bayes (and, in general, Bayesian networks)</li>
  </ul>
  
  <p>For a more complete list of such techniques and models, have a look at the online book <a href="https://christophm.github.io/interpretable-ml-book/index.html" rel="nofollow noreferrer">Interpretable Machine Learning: A Guide for Making Black Box Models Explainable</a>, by Christoph Molnar, which attempts to categorise and present the main XAI techniques.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-15T23:07:47.740Z
Answers.LastActivityDate: 2019-06-30T15:07:40.397Z
Answers.LastEditDate: 2019-06-30T15:07:40.397Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Jacques Pitrat's last book  <a href="https://www.kobo.com/us/en/ebook/artificial-beings" rel="nofollow noreferrer"><em>Artificial Beings
  The Conscience of a Conscious Machine</em></a>  has some interesting chapters (§2, §3, §5, §6, §7) related to the questions of explanations and meta-explanations in AI systems.</p>
  <p>It describes a reflexive, &quot;expert system&quot; like, meta-knowledge based approach to explainable AI</p>
  <p>You could also read his papers
  <a href="https://www.sciencedirect.com/science/article/abs/pii/0167739X96000118" rel="nofollow noreferrer"><em>Implementation of a reflective system</em></a> (1996)
  and <a href="http://jacques.pitrat.pagesperso-orange.fr/A%20Step%20toward%20an%20Artificial%20AI%20Scientist.pdf" rel="nofollow noreferrer"><em>A Step toward an Artificial AI Scientist</em></a> online (there could be a typo in it: &quot;pile&quot; is the French word for stack, including the call stack).</p>
  <p>You might also look into J.Pitrat's <a href="http://bootstrappingartificialintelligence.fr/WordPress3/" rel="nofollow noreferrer">blog</a> and into the ongoing <a href="http://refpersys.org/" rel="nofollow noreferrer">RefPerSys</a> project (it is in November 2020).</p>
  <p>PS. J.Pitrat (born in 1934) passed away in October 2019. French readers could see <a href="https://afia.asso.fr/journee-hommage-j-pitrat/" rel="nofollow noreferrer">this</a>. His blog might disappear in a few months.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-16T16:56:56.983Z
Answers.LastActivityDate: 2020-11-19T12:26:22.510Z
Answers.LastEditDate: 2020-11-19T12:26:22.510Z
Answers.UserDisplayName: Basile Starynkevitch
Body: <p><a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="noreferrer">Explainable artificial intelligence (XAI)</a> is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in particular, machine learning algorithms and models, especially black-box ones, such as artificial neural networks, so that these can also be adopted in areas, like healthcare, where the interpretability and understanding of the results (e.g. classifications) are required.</p>
  <p>Which XAI techniques are there?</p>
  <p>If there are many, to avoid making this question too broad, you can just provide a few examples (the most famous or effective ones), and, for people interested in more techniques and details, you can also provide one or more references/surveys/books that go into the details of XAI. The idea of this question is that people could easily find one technique that they could study to understand what XAI really is or how it can be approached.</p>
  
Tags: <reference-request><ethics><explainable-ai>
Title: Which explainable artificial intelligence techniques are there?
UserDisplayName: nbro
ThreadText: # Which explainable artificial intelligence techniques are there?
  
  (Asked by: username_0 on 2019-06-15)
  
  [Explainable artificial intelligence (XAI)](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence) is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in particular, machine learning algorithms and models, especially black-box ones, such as artificial neural networks, so that these can also be adopted in areas, like healthcare, where the interpretability and understanding of the results (e.g. classifications) are required.
  
  Which XAI techniques are there?
  
  If there are many, to avoid making this question too broad, you can just provide a few examples (the most famous or effective ones), and, for people interested in more techniques and details, you can also provide one or more references/surveys/books that go into the details of XAI. The idea of this question is that people could easily find one technique that they could study to understand what XAI really is or how it can be approached.
  
  **Tags:** reference-request, ethics, explainable-ai
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-10-18. Score: 31
  
  *Explainable AI* and *model interpretability* are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks, plus the necessity of algorithmic fairness & accountability.
  
  Here are some state of the art algorithms and approaches, together with implementations and frameworks.
  
  ---
  
  **Model-agnostic approaches**
  
  * LIME: Local Interpretable Model-agnostic Explanations ([paper](https://arxiv.org/abs/1602.04938), [code](https://github.com/marcotcr/lime), [blog post](https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime), [R port](https://cran.r-project.org/web/packages/lime/index.html))
  * SHAP: A Unified Approach to Interpreting Model Predictions ([paper](https://arxiv.org/abs/1705.07874), [Python package](https://github.com/slundberg/shap), [R package](https://github.com/redichh/ShapleyR)). GPU implementation for tree models by NVIDIA using RAPIDS - GPUTreeShap ([paper](https://arxiv.org/abs/2010.13972), [code](https://github.com/rapidsai/gputreeshap), [blog post](https://developer.nvidia.com/blog/explaining-and-accelerating-machine-learning-for-loan-delinquencies/))
  * Anchors: High-Precision Model-Agnostic Explanations ([paper](https://homes.cs.washington.edu/%7Emarcotcr/aaai18.pdf), [authors' Python code](https://github.com/marcotcr/anchor), [Java](https://github.com/viadee/javaAnchorExplainer) implementation)
  * Diverse Counterfactual Explanations (DiCE) by Microsoft ([paper](https://arxiv.org/abs/1905.07697), [code](https://github.com/microsoft/dice), [blog post](https://www.microsoft.com/en-us/research/blog/open-source-library-provides-explanation-for-machine-learning-through-diverse-counterfactuals/))
  * [Black Box Auditing](https://arxiv.org/abs/1602.07043) and [Certifying and Removing Disparate Impact](https://arxiv.org/abs/1412.3756) (authors' [Python code](https://github.com/algofairness/BlackBoxAuditing))
  * FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs ([blog post](http://blog.fastforwardlabs.com/2017/03/09/fairml-auditing-black-box-predictive-models.html), [paper](https://arxiv.org/abs/1611.04967), [code](https://github.com/adebayoj/fairml))
  
  SHAP seems to enjoy [high popularity](https://twitter.com/christophmolnar/status/1162460768631177216?s=11) among practitioners; the method has firm theoretical foundations on co-operational game theory (Shapley values), and it has in a great degree integrated the LIME approach under a common framework. Although model-agnostic, specific & efficient implementations are available for neural networks ([DeepExplainer](https://shap.readthedocs.io/en/latest/generated/shap.DeepExplainer.html)) and tree ensembles ([TreeExplainer](https://shap.readthedocs.io/en/latest/generated/shap.TreeExplainer.html), [paper](https://www.nature.com/articles/s42256-019-0138-9.epdf?shared_access_token=RCYPTVkiECUmc0CccSMgXtRgN0jAjWel9jnR3ZoTv0O81kV8DqPb2VXSseRmof0Pl8YSOZy4FHz5vMc3xsxcX6uT10EzEoWo7B-nZQAHJJvBYhQJTT1LnJmpsa48nlgUWrMkThFrEIvZstjQ7Xdc5g%3D%3D)).
  
  ---
  
  **Neural network approaches** (mostly, but not exclusively, for computer vision models)
  
  * The Layer-wise Relevance Propagation (LRP) toolbox for neural networks ([2015 paper @ PLoS ONE](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140), [2016 paper @ JMLR](http://www.jmlr.org/papers/v17/15-618.html), [project page](http://heatmapping.org/), [code](https://github.com/sebastian-lapuschkin/lrp_toolbox), [TF Slim wrapper](https://github.com/VigneshSrinivasan10/interprettensor))
  * Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization ([paper](https://arxiv.org/abs/1610.02391), authors' [Torch code](https://github.com/ramprs/grad-cam), [Tensorflow code](https://github.com/Ankush96/grad-cam.tensorflow), [PyTorch code](https://github.com/meliketoy/gradcam.pytorch), yet another [Pytorch implementation](https://github.com/jacobgil/pytorch-grad-cam), Keras [example notebook](http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb), Coursera [Guided Project](https://www.coursera.org/projects/scene-classification-gradcam))
  * Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method ([paper](https://arxiv.org/abs/2008.02312), [code](https://github.com/Fu0511/XGrad-CAM))
  * SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability ([paper](https://arxiv.org/abs/1706.05806), [code](https://github.com/google/svcca), [Google blog post](https://ai.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html))
  * TCAV: Testing with Concept Activation Vectors ([ICML 2018 paper](https://arxiv.org/abs/1711.11279), [Tensorflow code](https://github.com/tensorflow/tcav))
  * Integrated Gradients ([paper](https://arxiv.org/abs/1703.01365), [code](https://github.com/ankurtaly/Integrated-Gradients), [Tensorflow tutorial](https://www.tensorflow.org/tutorials/interpretability/integrated_gradients), [independent implementations](https://paperswithcode.com/paper/axiomatic-attribution-for-deep-networks#code))
  * Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL ([project page](http://netdissect.csail.mit.edu/), [Caffe code](https://github.com/CSAILVision/NetDissect), [PyTorch port](https://github.com/CSAILVision/NetDissect-Lite))
  * GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL ([project page](https://gandissect.csail.mit.edu/), with links to paper & code)
  * Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions ([paper](https://arxiv.org/abs/1811.08011), [code](https://github.com/gudovskiy/e2x))
  * Transparecy-by-Design (TbD) networks ([paper](https://arxiv.org/abs/1803.05268), [code](https://github.com/davidmascharka/tbd-nets), [demo](https://mybinder.org/v2/gh/davidmascharka/tbd-nets/binder?filepath=full-vqa-example.ipynb))
  * [Distilling a Neural Network Into a Soft Decision Tree](https://arxiv.org/abs/1711.09784), a 2017 paper by Geoff Hinton, with various independent [PyTorch implementations](https://paperswithcode.com/paper/distilling-a-neural-network-into-a-soft)
  * Understanding Deep Networks via Extremal Perturbations and Smooth Masks ([paper](https://arxiv.org/abs/1910.08485)), implemented in [TorchRay](https://github.com/facebookresearch/torchray) (see below)
  * Understanding the Role of Individual Units in a Deep Neural Network ([preprint](https://arxiv.org/abs/2009.05041), [2020 paper @ PNAS](https://www.pnas.org/content/early/2020/08/31/1907375117), [code](https://github.com/davidbau/dissect), [project page](https://dissect.csail.mit.edu/))
  * GNNExplainer: Generating Explanations for Graph Neural Networks ([paper](https://proceedings.neurips.cc/paper/2019/file/d80b7040b773199015de6d3b4293c8ff-Paper.pdf), [code](https://github.com/RexYing/gnn-model-explainer))
  * Benchmarking Deep Learning Interpretability in Time Series Predictions ([paper](https://proceedings.neurips.cc//paper/2020/file/47a3893cc405396a5c30d91320572d6d-Paper.pdf) @ NeurIPS 2020, [code](https://github.com/ayaabdelsalam91/TS-Interpretability-Benchmark) utilizing [Captum](https://captum.ai))
  * Concept Whitening for Interpretable Image Recognition ([paper](https://www.nature.com/articles/s42256-020-00265-z), [preprint](https://arxiv.org/abs/2002.01650), [code](https://github.com/zhiCHEN96/ConceptWhitening))
  
  ---
  
  **Libraries & frameworks**
  
  As interpretability moves toward the mainstream, there are already frameworks and toolboxes that incorporate more than one of the algorithms and techniques mentioned and linked above; here is a partial list:
  
  * The ELI5 Python library ([code](https://github.com/TeamHG-Memex/eli5), [documentation](https://eli5.readthedocs.io/en/latest/))
  * DALEX - moDel Agnostic Language for Exploration and eXplanation ([homepage](https://modeloriented.github.io/DALEX/), [code](https://github.com/ModelOriented/DALEX), [JMLR paper](https://www.jmlr.org/papers/volume19/18-416/18-416.pdf)), part of the [DrWhy.AI](https://modeloriented.github.io/DrWhy/) project
  * The What-If tool by Google, a feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code ([project page](https://pair-code.github.io/what-if-tool/), [code](https://github.com/pair-code/what-if-tool), [blog post](https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html))
  * The Language Interpretability Tool (LIT) by Google, a visual, interactive model-understanding tool for NLP models ([project page](https://pair-code.github.io/lit/), [code](https://github.com/PAIR-code/lit/), [blog post](https://ai.googleblog.com/2020/11/the-language-interpretability-tool-lit.html))
  * Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google ([code](https://github.com/tensorflow/lucid); papers: [Feature Visualization](https://distill.pub/2017/feature-visualization/), [The Building Blocks of Interpretability](https://distill.pub/2018/building-blocks/))
  * [TorchRay](https://github.com/facebookresearch/torchray) by Facebook, a PyTorch package implementing several visualization methods for deep CNNs
  * iNNvestigate Neural Networks ([code](https://github.com/albermax/innvestigate), [JMLR paper](https://jmlr.org/papers/v20/18-540.html))
  * tf-explain - interpretability methods as Tensorflow 2.0 callbacks ([code](https://github.com/sicara/tf-explain), [docs](https://tf-explain.readthedocs.io/en/latest/), [blog post](https://blog.sicara.com/tf-explain-interpretability-tensorflow-2-9438b5846e35))
  * InterpretML by Microsoft ([homepage](https://interpret.ml/), [code](https://github.com/Microsoft/interpret) still in alpha, [paper](https://arxiv.org/abs/1909.09223))
  * Captum by Facebook AI - model interpetability for Pytorch ([homepage](https://captum.ai), [code](https://github.com/pytorch/captum), [intro blog post](https://ai.facebook.com/blog/open-sourcing-captum-a-model-interpretability-library-for-pytorch/))
  * Skater, by Oracle ([code](https://github.com/oracle/Skater), [docs](https://oracle.github.io/Skater/))
  * Alibi, by SeldonIO ([code](https://github.com/SeldonIO/alibi), [docs](https://docs.seldon.io/projects/alibi/en/stable/))
  * AI Explainability 360, commenced by IBM and moved to the Linux Foundation ([homepage](https://ai-explainability-360.org/), [code](https://github.com/Trusted-AI/AIX360), [docs](https://aix360.readthedocs.io/en/latest/), [IBM Bluemix](http://aix360.mybluemix.net/), [blog post](https://www.ibm.com/blogs/research/2019/08/ai-explainability-360/))
  * Ecco: explaining transformer-based NLP models using interactive visualizations ([homepage](https://www.eccox.io/), [code](https://github.com/jalammar/ecco), [article](https://jalammar.github.io/explaining-transformers/)).
  * Recipes for Machine Learning Interpretability in H2O Driverless AI ([repo](https://github.com/h2oai/driverlessai-recipes/tree/master/explainers))
  
  ---
  
  **Reviews & general papers**
  
  * [A Survey of Methods for Explaining Black Box Models](https://dl.acm.org/doi/10.1145/3236009) (2018, ACM Computing Surveys)
  * [Definitions, methods, and applications in interpretable machine learning](https://www.pnas.org/content/116/44/22071) (2019, PNAS)
  * [Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead](https://www.nature.com/articles/s42256-019-0048-x) (2019, Nature Machine Intelligence, [preprint](https://arxiv.org/abs/1811.10154))
  * [Machine Learning Interpretability: A Survey on Methods and Metrics](https://www.mdpi.com/2079-9292/8/8/832) (2019, Electronics)
  * [Principles and Practice of Explainable Machine Learning](https://arxiv.org/abs/2009.11698) (2020, preprint)
  * [Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges](https://arxiv.org/abs/2010.09337) (keynote at [2020 ECML XKDD workshop](https://kdd.isti.cnr.it/xkdd2020/) by Christoph Molnar, [video & slides](https://slideslive.com/38933066/interpretable-machine-learning-state-of-the-art-and-challenges))
  * [Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI](https://www.sciencedirect.com/science/article/pii/S1566253519308103) (2020, Information Fusion)
  * [Counterfactual Explanations for Machine Learning: A Review](https://arxiv.org/abs/2010.10596) (2020, preprint, [critique](https://twitter.com/yudapearl/status/1321038352355790848?s=11) by Judea Pearl)
  * [Interpretability 2020](https://ff06-2020.fastforwardlabs.com/), an applied research report by Cloudera Fast Forward, updated regularly
  * [Interpreting Predictions of NLP Models](https://github.com/Eric-Wallace/interpretability-tutorial-emnlp2020) (EMNLP 2020 tutorial)
  * Explainable NLP Datasets ([site](https://exnlpdatasets.github.io/), [preprint](https://arxiv.org/abs/2102.12060), [highlights](https://twitter.com/sarahwiegreffe/status/1365057786250489858))
  * [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://arxiv.org/abs/2103.11251)
  
  ---
  
  **eBooks** (available online)
  
  * [Interpretable Machine Learning](https://christophm.github.io/interpretable-ml-book/), by Christoph Molnar, with [R code](https://github.com/christophM/iml) available
  * [Explanatory Model Analysis](https://pbiecek.github.io/ema/), by DALEX creators Przemyslaw Biecek and Tomasz Burzykowski, with both R & Python code snippets
  * [An Introduction to Machine Learning Interpretability](https://www.h2o.ai/wp-content/uploads/2019/08/An-Introduction-to-Machine-Learning-Interpretability-Second-Edition.pdf) (2nd ed. 2019), by H2O
  
  ---
  
  **Online courses & tutorials**
  
  * [Machine Learning Explainability](https://www.kaggle.com/learn/machine-learning-explainability), Kaggle tutorial
  * [Explainable AI: Scene Classification and GradCam Visualization](https://www.coursera.org/projects/scene-classification-gradcam), Coursera guided project
  * [Explainable Machine Learning with LIME and H2O in R](http://Explainable%20Machine%20Learning%20with%20LIME%20and%20H2O%20in%20R), Coursera guided project
  * [Interpretability and Explainability in Machine Learning](https://interpretable-ml-class.github.io/), Harvard COMPSCI 282BR
  
  ---
  
  **Other resources**
  
  * [explained.ai](https://explained.ai/) blog
  * A [Twitter thread](https://twitter.com/ledell/status/995930308947140608), linking to several interpretation tools available for R
  * A whole bunch of resources in the [Awesome Machine Learning Interpetability](https://github.com/jphall663/awesome-machine-learning-interpretability) repo
  * The online comic book (!) [*The Hitchhiker's Guide to Responsible Machine Learning*](https://betaandbit.github.io/RML/#p=1), by the team behind the textbook *Explanatory Model Analysis* and the DALEX package mentioned above ([blog post](https://medium.com/responsibleml/the-hitchhikers-guide-to-responsible-machine-learning-d9179a047319) and [backstage](https://medium.com/responsibleml/backstage-the-hitchhikers-guide-to-responsible-machine-learning-3a8735a0f8bb))
  
  > **Comments:**
  > **username_2**: Hey.  According to the paper, GNNEXPLAINER may be model-agnostic rather than neural-network-based. I paraphrase from the paper "Here we propose GNNEXPLAINER, the first general, *model-agnostic* approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task."  
  
  ---
  
  ### Answer by username_0 on 2019-06-15. Score: 5
  
  There are a few [XAI techniques](https://christophm.github.io/interpretable-ml-book/agnostic.html) that are (partially) agnostic to the model to be interpreted
  
  * Layer-wise relevance propagation (LRP), introduced in [On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140) (2015)
  * Local Interpretable Model-agnostic Explanations (LIME), introduced in ["Why Should I Trust You?" Explaining the Predictions of Any Classifier](https://arxiv.org/pdf/1602.04938v1.pdf) (2016)
  * Model Agnostic Contrastive Explanations Method (MACEM), introduced in [Model Agnostic Contrastive Explanations for Structured Data](https://arxiv.org/pdf/1906.00117.pdf) (2019)
  
  There are also ML models that are not considered black boxes and that are thus [more interpretable than black boxes](https://christophm.github.io/interpretable-ml-book/simple.html), such as
  
  * linear models (e.g. linear regression)
  * decision trees
  * naive Bayes (and, in general, Bayesian networks)
  
  For a more complete list of such techniques and models, have a look at the online book [Interpretable Machine Learning: A Guide for Making Black Box Models Explainable](https://christophm.github.io/interpretable-ml-book/index.html), by Christoph Molnar, which attempts to categorise and present the main XAI techniques.
  
  
  ---
  
  ### Answer by username_3 on 2020-11-16. Score: 0
  
  Jacques Pitrat's last book [*Artificial Beings
  The Conscience of a Conscious Machine*](https://www.kobo.com/us/en/ebook/artificial-beings) has some interesting chapters (§2, §3, §5, §6, §7) related to the questions of explanations and meta-explanations in AI systems.
  
  It describes a reflexive, "expert system" like, meta-knowledge based approach to explainable AI
  
  You could also read his papers
  [*Implementation of a reflective system*](https://www.sciencedirect.com/science/article/abs/pii/0167739X96000118) (1996)
  and [*A Step toward an Artificial AI Scientist*](http://jacques.pitrat.pagesperso-orange.fr/A%20Step%20toward%20an%20Artificial%20AI%20Scientist.pdf) online (there could be a typo in it: "pile" is the French word for stack, including the call stack).
  
  You might also look into J.Pitrat's [blog](http://bootstrappingartificialintelligence.fr/WordPress3/) and into the ongoing [RefPerSys](http://refpersys.org/) project (it is in November 2020).
  
  PS. J.Pitrat (born in 1934) passed away in October 2019. French readers could see [this](https://afia.asso.fr/journee-hommage-j-pitrat/). His blog might disappear in a few months.
  
  
  ---
  
  