Answers.Body: <p>You seem to be wanting some description of the 'style' of an image. </p>
  
  <p>To make that work in general, I'd guess that would actually require quite a lot of pre-processing to present 'texture elements' (rather than pixels) as the basic features. </p>
  
  <p>This is quite speculative, but one approach might be to use <a href="https://en.wikipedia.org/wiki/Iterated_function_system" rel="nofollow">Iterated Function Systems</a> as a means of extracting these.</p>
  
  <p>Whether 'spatial adjacency' (and hence CNN) is then the best approach to make higher-level decisions about these elements is (AFAIK) a matter for experiment.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-18T20:40:24.213Z
Answers.LastActivityDate: 2016-08-18T20:40:24.213Z
Answers.UserDisplayName: NietzscheanAI
Answers.Body: <p>If I look at the image, I can kind of see a monocle as <em>part</em> of the image. So one part of this is that the classifier is ignoring much of the image. This could be called a lack of "completeness", in the sense used <a href="http://www.wisdom.weizmann.ac.il/~vision/VisualSummary.html" rel="nofollow">here</a> (a computer vision paper on image summarization).</p>
  
  <p>One way to discover these sorts of failure modes is <a href="https://plus.google.com/+ResearchatGoogle/posts/QoFzqQBeANN" rel="nofollow">adversarial images</a>, which are optimized to fool a given image classifier. Building on this, the idea of <em>adversarial training</em> is to simultaneously train competing "machines", one trying to synthesize data, the other trying to find weaknesses in the first one.</p>
  
  <p>Also check this page: <a href="https://code.facebook.com/posts/1587249151575490/a-path-to-unsupervised-learning-through-adversarial-networks/" rel="nofollow">A path to unsupervised learning through adversarial networks</a>, for further information about adversarial training.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-23T00:09:27.470Z
Answers.LastActivityDate: 2016-08-23T00:14:37.027Z
Answers.LastEditDate: 2016-08-23T00:14:37.027Z
Answers.UserDisplayName: GeoMatt22
Answers.Body: <p>Wolfram's image id system is specifically meant to figure out what the image is depicting, not the medium. </p>
  
  <p>To get what you want you'd simply have to create your own system where the training data is labeled by the medium rather than the content, and probably fiddle with it to pay more attention to texture and things as such as that. The neural net doesn't care which we want - it has no inherent bias. It just knows what it's been trained for.</p>
  
  <p>That's really all there is to it. It's all to do with the training labels and the focus of the system (e.g. a system that looks for edge patterns that form shapes, compared to a system that checks if the lines in the image are perfectly computer-generated straight and clean vs imperfect brush strokes vs spraypaint).</p>
  
  <p>Now, if you want me to tell you how to build that system, I'm not the right person to ask haha</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-19T23:40:48.100Z
Answers.LastActivityDate: 2016-08-19T23:46:11.350Z
Answers.LastEditDate: 2016-08-19T23:46:11.350Z
Answers.UserDisplayName: Avik Mohan
Body: <p>I've <a href="https://www.imageidentify.com/result/0lkzuttdxipub" rel="nofollow noreferrer">uploaded a picture</a> to Wolfram's ImageIdentify of graffiti on the wall, but it recognized it as 'monocle'. Secondary guesses were 'primate', 'hominid', and 'person', so not even close to 'graffiti' or 'painting'.</p>
  
  <p>Is it by design, or there are some <strong>methods to teach a convolutional neural network (CNN) to reason and be aware of a bigger picture context</strong> (like mentioned graffiti)? Currently it seems as if it's detecting literally <em>what is depicted in the image</em>, not <em>what the image actually is</em>.</p>
  
  <p><a href="https://i.sstatic.net/akquMm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/akquMm.png" alt="Wolfram&#39;s Image Identify: monocle/graffiti"></a></p>
  
  <p>This could be the same problem as mentioned <a href="https://ai.stackexchange.com/a/1533/8">here</a>, that DNN are:</p>
  
  <blockquote>
    <p>Learning to detect jaguars by matching the unique spots on their fur while ignoring the fact that they have four legs.<sup><a href="https://ai.stackexchange.com/a/1533/8">2015</a></sup></p>
  </blockquote>
  
  <p>If it's by design, maybe there is some better version of CNN that can perform better?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-18T17:50:01.503Z
Comments.Text: https://i.sstatic.net/qu8K3.png
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-18T18:05:00.597Z
Comments.Text: So obviously it still needs some work.
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-19T05:15:34.433Z
Comments.Text: Any ANN (or for that matter any ML model) learns only from the training data that was used to train it. If the training data doesn't have examples of "graffiti" then there is no way the model can learn that.
Comments.UserDisplayName: Ankur
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-23T20:28:49.450Z
Comments.Text: Related: http://ai.stackexchange.com/questions/1689/how-wolframs-image-identification-project-works
Comments.UserDisplayName: Mithical
Tags: <image-recognition><classification><convolutional-neural-networks>
Title: How to make convnets aware what the image actually is, not what is depicted on it?
UserDisplayName: kenorb
ThreadText: # How to make convnets aware what the image actually is, not what is depicted on it?
  
  (Asked by: username_0 on 2016-08-18)
  
  I've [uploaded a picture](https://www.imageidentify.com/result/0lkzuttdxipub) to Wolfram's ImageIdentify of graffiti on the wall, but it recognized it as 'monocle'. Secondary guesses were 'primate', 'hominid', and 'person', so not even close to 'graffiti' or 'painting'.
  
  Is it by design, or there are some **methods to teach a convolutional neural network (CNN) to reason and be aware of a bigger picture context** (like mentioned graffiti)? Currently it seems as if it's detecting literally *what is depicted in the image*, not *what the image actually is*.
  
  [![Wolfram's Image Identify: monocle/graffiti](https://i.sstatic.net/akquMm.png)](https://i.sstatic.net/akquMm.png)
  
  This could be the same problem as mentioned [here](https://ai.stackexchange.com/a/1533/8), that DNN are:
  
  > Learning to detect jaguars by matching the unique spots on their fur while ignoring the fact that they have four legs.[2015](https://ai.stackexchange.com/a/1533/8)
  
  If it's by design, maybe there is some better version of CNN that can perform better?
  
  **Tags:** image-recognition, classification, convolutional-neural-networks
  > **Comments:**
  > **username_1**: https://i.sstatic.net/qu8K3.png  
  > **username_1**: So obviously it still needs some work.  
  > **username_2**: Any ANN (or for that matter any ML model) learns only from the training data that was used to train it. If the training data doesn't have examples of "graffiti" then there is no way the model can learn that.  
  > **username_1**: Related: http://ai.stackexchange.com/questions/1689/how-wolframs-image-identification-project-works  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2016-08-19. Score: 2 | Accepted answer
  
  Wolfram's image id system is specifically meant to figure out what the image is depicting, not the medium.
  
  To get what you want you'd simply have to create your own system where the training data is labeled by the medium rather than the content, and probably fiddle with it to pay more attention to texture and things as such as that. The neural net doesn't care which we want - it has no inherent bias. It just knows what it's been trained for.
  
  That's really all there is to it. It's all to do with the training labels and the focus of the system (e.g. a system that looks for edge patterns that form shapes, compared to a system that checks if the lines in the image are perfectly computer-generated straight and clean vs imperfect brush strokes vs spraypaint).
  
  Now, if you want me to tell you how to build that system, I'm not the right person to ask haha
  
  
  ---
  
  ### Answer by username_3 on 2016-08-18. Score: 3
  
  You seem to be wanting some description of the 'style' of an image.
  
  To make that work in general, I'd guess that would actually require quite a lot of pre-processing to present 'texture elements' (rather than pixels) as the basic features.
  
  This is quite speculative, but one approach might be to use [Iterated Function Systems](https://en.wikipedia.org/wiki/Iterated_function_system) as a means of extracting these.
  
  Whether 'spatial adjacency' (and hence CNN) is then the best approach to make higher-level decisions about these elements is (AFAIK) a matter for experiment.
  
  
  ---
  
  ### Answer by username_4 on 2016-08-23. Score: 0
  
  If I look at the image, I can kind of see a monocle as *part* of the image. So one part of this is that the classifier is ignoring much of the image. This could be called a lack of "completeness", in the sense used [here](http://www.wisdom.weizmann.ac.il/~vision/VisualSummary.html) (a computer vision paper on image summarization).
  
  One way to discover these sorts of failure modes is [adversarial images](https://plus.google.com/+ResearchatGoogle/posts/QoFzqQBeANN), which are optimized to fool a given image classifier. Building on this, the idea of *adversarial training* is to simultaneously train competing "machines", one trying to synthesize data, the other trying to find weaknesses in the first one.
  
  Also check this page: [A path to unsupervised learning through adversarial networks](https://code.facebook.com/posts/1587249151575490/a-path-to-unsupervised-learning-through-adversarial-networks/), for further information about adversarial training.
  
  
  ---
  
  