Answers.Body: <p>Testing machine learning programs is quite different than testing traditional software.</p>
  <p>The main reason why this is the case is quite simple, if you're familiar with machine learning.</p>
  <p>ML programs are not just if statements and loops, but they are composed of <strong>models</strong>, which can even be black-box models, such as neural networks (i.e. it's difficult to interpret the function that they compute). These models are trained to approximate some unknown function, which is only partially described by some given <strong>data</strong>, which can also contain spurious/noisy information.</p>
  <p>For this reason, traditional testing techniques, such as <em>statement coverage</em>, are <strong>insufficient</strong> to fully &quot;test&quot; ML programs: e.g. even if all statements of your ML program are covered, your ML program can still fail, e.g. it may predict the wrong class for some previously unseen input. So, testing ML programs requires not only traditional testing techniques, but also other approaches that attempt to address the (un-)desirable behaviour (such as the <em>generalization</em>) of the models.</p>
  <p>Currently, I am also doing research on this specific topic, so I can say that there are already several people working on this topic, but the field is still quite immature and there are still many unsolved problems. It's also important to note that the general &quot;testing problem&quot;, even for traditional software, is not yet solved.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T15:48:40.640Z
Answers.LastActivityDate: 2021-04-05T15:48:40.640Z
Answers.UserDisplayName: nbro
Body: <p>The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see <a href="https://stats.stackexchange.com/q/442773/103153">this question</a> and <a href="https://stats.stackexchange.com/q/507376/103153">this one</a>).</p>
  <p>What are the differences between the two?</p>
  
Tags: <machine-learning><testing>
Title: What are the differences in testing between traditional software and artificial intelligence?
UserDisplayName: Lerner Zhang
ThreadText: # What are the differences in testing between traditional software and artificial intelligence?
  
  (Asked by: username_0 on 2021-04-05)
  
  The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see [this question](https://stats.stackexchange.com/q/442773/103153) and [this one](https://stats.stackexchange.com/q/507376/103153)).
  
  What are the differences between the two?
  
  **Tags:** machine-learning, testing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-05. Score: 1
  
  Testing machine learning programs is quite different than testing traditional software.
  
  The main reason why this is the case is quite simple, if you're familiar with machine learning.
  
  ML programs are not just if statements and loops, but they are composed of **models**, which can even be black-box models, such as neural networks (i.e. it's difficult to interpret the function that they compute). These models are trained to approximate some unknown function, which is only partially described by some given **data**, which can also contain spurious/noisy information.
  
  For this reason, traditional testing techniques, such as *statement coverage*, are **insufficient** to fully "test" ML programs: e.g. even if all statements of your ML program are covered, your ML program can still fail, e.g. it may predict the wrong class for some previously unseen input. So, testing ML programs requires not only traditional testing techniques, but also other approaches that attempt to address the (un-)desirable behaviour (such as the *generalization*) of the models.
  
  Currently, I am also doing research on this specific topic, so I can say that there are already several people working on this topic, but the field is still quite immature and there are still many unsolved problems. It's also important to note that the general "testing problem", even for traditional software, is not yet solved.
  
  
  ---
  
  