Answers.Body: <p>I think you are looking for the field known as <strong>explainable artificial intelligence</strong>. The book <a href="https://christophm.github.io/interpretable-ml-book/" rel="nofollow noreferrer">Interpretable Machine Learning: A Guide for Making Black Box Models Explainable</a> will surely help you to understand the issues and existing techniques. See also the following question <a href="https://ai.stackexchange.com/q/12870/2444">Which explainable artificial intelligence techniques are there?</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T21:45:32.013Z
Answers.LastActivityDate: 2020-07-29T21:45:32.013Z
Answers.UserDisplayName: nbro
Answers.Body: <p>I have one concept, which allows to structure knowledge gathered by a ML system in both categorical and algorithms-like structure.</p>
  
  <p>The key idea here is that we contain in our minds some network-like structure in our minds, which helping us not only to classify some data, but also to form an abstract meaning about text or message.</p>
  
  <p>Other than that, in opposite to our minds, it looks like <a href="https://stackoverflow.com/questions/31268796/concept-based-text-summarization-abstraction/31272242#31272242">this</a> solution can produce these structures in readable for us forms.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-01T15:05:43.957Z
Answers.LastActivityDate: 2019-11-01T15:05:43.957Z
Answers.UserDisplayName: Stepan Novikov
Answers.Body: <p>We actually can explain how the machine answers our questions. For example, we know how the ML works for the case of image recognition. See the following paper by Chris Olah:</p>
  <p><a href="https://distill.pub/2017/feature-visualization/" rel="nofollow noreferrer">https://distill.pub/2017/feature-visualization/</a>
  and also
  <a href="https://distill.pub/2018/building-blocks/" rel="nofollow noreferrer">https://distill.pub/2018/building-blocks/</a></p>
  <p>By feature visualization, we know that the machine sees at each layer and how it decides using given information. As the images are visual it is easy to &quot;visualize&quot; the features but of course, interpreting inner workings of a Neural Network for other kinds of data (such as financial data, for example) would be much more difficult I presume. But still, we can understand how the initial data is weighted or how the intermediate features are calculated.</p>
  <p>Given that, I highly doubt the reasoning of a machine can be taught to a human. It is far too complex to be learned. We can easily understand one or two dimensional functions, but the data we use in machine learning problems usually have hundreds of dimensions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-02T14:36:59.137Z
Answers.Text: When human engineers design a system, say a moon rocket, of course computers are used to speed the calculations, but the engineers *could* "do it by hand".  A moon rocket can contain hundreds of thousands of components.  Because the design is hierarchical, it is possible to train an engineer to do any "local" or "global" part of the design process, and understand what he's doing and why it it necessary.  A human being could be trained to do any small enough part of what a machine learning system does, but unless the ML system is hierarchical, it's hard for humans to understand the "why".
Answers.UserDisplayName: S. McGrew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-02T09:00:40.710Z
Answers.LastActivityDate: 2021-01-06T13:25:46.587Z
Answers.LastEditDate: 2021-01-06T13:25:46.587Z
Answers.UserDisplayName: serali
Body: <p>The thing about machine learning (ML) that worries me is that &quot;knowledge&quot; acquired in ML is hidden: we usually can't explain the criteria or methods used by the machine to provide an answer when we ask it a question.</p>
  <p>It's as if we asked an expert financial analyst for advice and he/she replied, &quot;Invest in X&quot;; then when we asked &quot;Why?&quot;, the analyst answered, &quot;Because I have a feeling that's the right thing for you to do.&quot;  It makes us dependent on the analyst.</p>
  <p>Surely there are some researchers trying to find ways for ML systems to encapsulate and refine their &quot;knowledge&quot; into a form that can then be taught to a human or encoded into a much simpler machine.  Who, if any, are working on that?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-29T21:42:43.433Z
Comments.Text: Please, next time when you ask a question, put your question also on the title. I think that this helps to attract more people because they can have an idea of what your post is about. Btw, you're looking for "explainable artificial intelligence".
Comments.UserDisplayName: nbro
Tags: <machine-learning><reference-request><knowledge-representation><explainable-ai>
Title: Who is working on explaining the knowledge encoded into machine learning models?
UserDisplayName: S. McGrew
ThreadText: # Who is working on explaining the knowledge encoded into machine learning models?
  
  (Asked by: username_0 on 2019-11-01)
  
  The thing about machine learning (ML) that worries me is that "knowledge" acquired in ML is hidden: we usually can't explain the criteria or methods used by the machine to provide an answer when we ask it a question.
  
  It's as if we asked an expert financial analyst for advice and he/she replied, "Invest in X"; then when we asked "Why?", the analyst answered, "Because I have a feeling that's the right thing for you to do." It makes us dependent on the analyst.
  
  Surely there are some researchers trying to find ways for ML systems to encapsulate and refine their "knowledge" into a form that can then be taught to a human or encoded into a much simpler machine. Who, if any, are working on that?
  
  **Tags:** machine-learning, reference-request, knowledge-representation, explainable-ai
  > **Comments:**
  > **username_1**: Please, next time when you ask a question, put your question also on the title. I think that this helps to attract more people because they can have an idea of what your post is about. Btw, you're looking for "explainable artificial intelligence".  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-07-29. Score: 0
  
  I think you are looking for the field known as **explainable artificial intelligence**. The book [Interpretable Machine Learning: A Guide for Making Black Box Models Explainable](https://christophm.github.io/interpretable-ml-book/) will surely help you to understand the issues and existing techniques. See also the following question [Which explainable artificial intelligence techniques are there?](https://ai.stackexchange.com/q/12870/2444).
  
  
  ---
  
  ### Answer by username_2 on 2019-11-01. Score: 0
  
  I have one concept, which allows to structure knowledge gathered by a ML system in both categorical and algorithms-like structure.
  
  The key idea here is that we contain in our minds some network-like structure in our minds, which helping us not only to classify some data, but also to form an abstract meaning about text or message.
  
  Other than that, in opposite to our minds, it looks like [this](https://stackoverflow.com/questions/31268796/concept-based-text-summarization-abstraction/31272242#31272242) solution can produce these structures in readable for us forms.
  
  
  ---
  
  ### Answer by username_3 on 2019-11-02. Score: 0
  
  We actually can explain how the machine answers our questions. For example, we know how the ML works for the case of image recognition. See the following paper by Chris Olah:
  
  <https://distill.pub/2017/feature-visualization/>
  and also
  <https://distill.pub/2018/building-blocks/>
  
  By feature visualization, we know that the machine sees at each layer and how it decides using given information. As the images are visual it is easy to "visualize" the features but of course, interpreting inner workings of a Neural Network for other kinds of data (such as financial data, for example) would be much more difficult I presume. But still, we can understand how the initial data is weighted or how the intermediate features are calculated.
  
  Given that, I highly doubt the reasoning of a machine can be taught to a human. It is far too complex to be learned. We can easily understand one or two dimensional functions, but the data we use in machine learning problems usually have hundreds of dimensions.
  
  > **Comments:**
  > **username_0**: When human engineers design a system, say a moon rocket, of course computers are used to speed the calculations, but the engineers *could* "do it by hand".  A moon rocket can contain hundreds of thousands of components.  Because the design is hierarchical, it is possible to train an engineer to do any "local" or "global" part of the design process, and understand what he's doing and why it it necessary.  A human being could be trained to do any small enough part of what a machine learning system does, but unless the ML system is hierarchical, it's hard for humans to understand the "why".  
  
  ---
  
  