Answers.Body: <p>One of the <em>Pinterest's</em> white paper about <a href="https://engineering.pinterest.com/sites/engineering/files/article/fields/field_image/human-curation-convnets%20(1).pdf" rel="nofollow noreferrer">Human Curation and Convnets powering item-to-item recommendations</a><sup><a href="https://arxiv.org/abs/1511.04003" rel="nofollow noreferrer">arxiv</a></sup> describes implementation of convolutional neural network (CNN) based visual features (<a href="http://www.robots.ox.ac.uk/~vgg/" rel="nofollow noreferrer">VGG</a><sup><a href="https://arxiv.org/abs/1405.3531" rel="nofollow noreferrer">2014</a></sup>, <a href="http://arxiv.org/abs/1506.01497" rel="nofollow noreferrer">Faster R-CNN</a>). This demonstrates the effectiveness of it (such image or object representations) which can improve user engagement. The visual features are computed using the process described in the <a href="http://arxiv.org/abs/1505.07647" rel="nofollow noreferrer">previous study about visual search at <em>Pinterest</em></a> and can be used for more targeted features to be computed for related pins.</p>
  
  <p>Here are the examples of detected visual objects from Pinterest's object detection pipeline:</p>
  
  <p><a href="https://i.sstatic.net/q1Tvr.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/q1Tvr.jpg" alt="Fig. 6. Examples of detected visual objects from Pinterest’s object detection pipeline. Detection of objects allows for more targeted visual features to be computed for Related Pins"></a></p>
  
  <p><sup>Image source: <a href="https://engineering.pinterest.com/sites/engineering/files/article/fields/field_image/human-curation-convnets%20(1).pdf" rel="nofollow noreferrer">Human Curation and Convnets: Powering
  Item-to-Item Recommendations on Pinterest</a>, Page 4, Fig. 6</sup></p>
  
  <p>The images are categorized by using dominant visual objects (individual objects seen in the image which passes a confidence threshold in Faster R-CNN) using fine-tuned VGG reranking variant. This allows Pinterest to introduce features such real-time recommendations for the users.</p>
  
  <p>Check also this blog entry: <a href="https://engineering.pinterest.com/blog/building-scalable-machine-vision-pipeline" rel="nofollow noreferrer">Building a scalable machine vision pipeline</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-18T17:03:13.773Z
Answers.LastActivityDate: 2016-08-23T02:04:02.893Z
Answers.LastEditDate: 2016-08-23T02:04:02.893Z
Answers.UserDisplayName: kenorb
Body: <p>According to this <a href="http://mashable.com/2014/01/06/pinterest-acquires-visualgraph/" rel="nofollow">article</a>, Pinterest acquired VisualGraph, an image recognition and visual search technology startup.</p>
  
  <p>How does Pinterest apply VisualGraph technology for machine vision, image recognition and visual search in order to classify the images?</p>
  
  <p>In short, how do they predict the image categories? Based on what features?</p>
  
Tags: <image-recognition><classification><computer-vision>
Title: How does Pinterest decipher what's on unmarked pictures and categorize them?
UserDisplayName: kenorb
ThreadText: # How does Pinterest decipher what's on unmarked pictures and categorize them?
  
  (Asked by: username_0 on 2016-08-18)
  
  According to this [article](http://mashable.com/2014/01/06/pinterest-acquires-visualgraph/), Pinterest acquired VisualGraph, an image recognition and visual search technology startup.
  
  How does Pinterest apply VisualGraph technology for machine vision, image recognition and visual search in order to classify the images?
  
  In short, how do they predict the image categories? Based on what features?
  
  **Tags:** image-recognition, classification, computer-vision
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2016-08-18. Score: 4 | Accepted answer
  
  One of the *Pinterest's* white paper about [Human Curation and Convnets powering item-to-item recommendations](https://engineering.pinterest.com/sites/engineering/files/article/fields/field_image/human-curation-convnets%20(1).pdf)[arxiv](https://arxiv.org/abs/1511.04003) describes implementation of convolutional neural network (CNN) based visual features ([VGG](http://www.robots.ox.ac.uk/~vgg/)[2014](https://arxiv.org/abs/1405.3531), [Faster R-CNN](http://arxiv.org/abs/1506.01497)). This demonstrates the effectiveness of it (such image or object representations) which can improve user engagement. The visual features are computed using the process described in the [previous study about visual search at *Pinterest*](http://arxiv.org/abs/1505.07647) and can be used for more targeted features to be computed for related pins.
  
  Here are the examples of detected visual objects from Pinterest's object detection pipeline:
  
  [![Fig. 6. Examples of detected visual objects from Pinterest’s object detection pipeline. Detection of objects allows for more targeted visual features to be computed for Related Pins](https://i.sstatic.net/q1Tvr.jpg)](https://i.sstatic.net/q1Tvr.jpg)
  
  Image source: [Human Curation and Convnets: Powering
  Item-to-Item Recommendations on Pinterest](https://engineering.pinterest.com/sites/engineering/files/article/fields/field_image/human-curation-convnets%20(1).pdf), Page 4, Fig. 6
  
  The images are categorized by using dominant visual objects (individual objects seen in the image which passes a confidence threshold in Faster R-CNN) using fine-tuned VGG reranking variant. This allows Pinterest to introduce features such real-time recommendations for the users.
  
  Check also this blog entry: [Building a scalable machine vision pipeline](https://engineering.pinterest.com/blog/building-scalable-machine-vision-pipeline).
  
  
  ---
  
  