Answers.Body: <p>I believe you may want to use a <a href="https://arxiv.org/pdf/1202.3732.pdf" rel="nofollow noreferrer">Sum Product Network</a> for this task. SPNs are the state-of-the-art approach for face completion, and there are several more <a href="https://arxiv.org/pdf/1902.06155.pdf" rel="nofollow noreferrer">recent papers</a> on this topic since the original above.</p>
  
  <p>Importantly, the SPN paper also covers other approaches that work well for this task. If lower-resolution results are acceptable for your task, PCA with 100 or more components works surprisingly well. If your dataset is very large, a nearest-neighbor approach can work decently too.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T20:11:42.487Z
Answers.Text: Thanks for your response. Please see my comment response to @ashenoy: could you please explain how to deal with some of the issues/concerns raised there about StyleGAN in terms of SPNs?
Answers.UserDisplayName: Geza Kerecsenyi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-16T21:48:04.700Z
Answers.LastActivityDate: 2019-11-16T21:48:04.700Z
Answers.UserDisplayName: John Doucette
Body: <p><a href="https://i.sstatic.net/5rN5a.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5rN5a.jpg" alt="image"></a></p>
  
  <p><sub>These images are handmade, not auto-generated like they will be in production. Apologies for inaccuracies in the graph overlay.</sub></p>
  
  <p>I am trying to build an AI like that displayed in the diagram: when given a training set of images with their corresponding node maps of face/nose posture, and an image with a missing <strong>section</strong> (just a gap) with a node map, I would like it to reconstruct the initial image. My thoughts immediately went to GANs for this, but after some searching, the closest I could find were:</p>
  
  <ul>
  <li>Face recreation without context/not filling gaps, just following pose (DeepFake)</li>
  <li>Filling gaps in images, but with no node reference</li>
  <li>Filling gaps from reference drawings/mappings, but with no way to provide sample images</li>
  </ul>
  
  <p>I would like to hear about any implementations of such an algorithm, if possible optimised for faces, and if none exists, I would like to hear of how I would go about altering the generator of the GAN to work with the context/gap-fill bit (e.g a paper which talks about this idea, but doesn't implement it). Any guidance on the NN that is best for this type of task is also appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-16T04:57:21.040Z
Comments.Text: Not come across an exact implementation of this but would recommend you check out the generator architecture of StyleGAN by NVLabs. Its a typical GAN mapping from latents to an image distribution but thought it might be helpful because they have a trained Generator on faces.
Comments.UserDisplayName: ashenoy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-20T20:10:22.923Z
Comments.Text: @ashenoy thanks for your comment, and apologies for not getting back: I wanted to make sure I understood the paper before giving a response. StyleGAN seems interesting, and as though it would have potential. I was wondering, would it be possible to hijack StyleGAN's discriminator to take the nodes into account? I think it would probably be easier to alter the _discriminator_, rather than telling the generator the node positions explicitly. However, this then just leaves the question of how could we make it do gap-filling? We somehow need to feed it a 'prompt' - how could this be initialised?
Comments.UserDisplayName: Geza Kerecsenyi
Tags: <ai-design><generative-adversarial-networks><generative-model><implementation><image-generation>
Title: Context-based gap-fill face posture-mapper GAN
UserDisplayName: Geza Kerecsenyi
ThreadText: # Context-based gap-fill face posture-mapper GAN
  
  (Asked by: username_0 on 2019-11-13)
  
  [![image](https://i.sstatic.net/5rN5a.jpg)](https://i.sstatic.net/5rN5a.jpg)
  
  These images are handmade, not auto-generated like they will be in production. Apologies for inaccuracies in the graph overlay.
  
  I am trying to build an AI like that displayed in the diagram: when given a training set of images with their corresponding node maps of face/nose posture, and an image with a missing **section** (just a gap) with a node map, I would like it to reconstruct the initial image. My thoughts immediately went to GANs for this, but after some searching, the closest I could find were:
  
  * Face recreation without context/not filling gaps, just following pose (DeepFake)
  * Filling gaps in images, but with no node reference
  * Filling gaps from reference drawings/mappings, but with no way to provide sample images
  
  I would like to hear about any implementations of such an algorithm, if possible optimised for faces, and if none exists, I would like to hear of how I would go about altering the generator of the GAN to work with the context/gap-fill bit (e.g a paper which talks about this idea, but doesn't implement it). Any guidance on the NN that is best for this type of task is also appreciated.
  
  **Tags:** ai-design, generative-adversarial-networks, generative-model, implementation, image-generation
  > **Comments:**
  > **username_1**: Not come across an exact implementation of this but would recommend you check out the generator architecture of StyleGAN by NVLabs. Its a typical GAN mapping from latents to an image distribution but thought it might be helpful because they have a trained Generator on faces.  
  > **username_0**: @ashenoy thanks for your comment, and apologies for not getting back: I wanted to make sure I understood the paper before giving a response. StyleGAN seems interesting, and as though it would have potential. I was wondering, would it be possible to hijack StyleGAN's discriminator to take the nodes into account? I think it would probably be easier to alter the _discriminator_, rather than telling the generator the node positions explicitly. However, this then just leaves the question of how could we make it do gap-filling? We somehow need to feed it a 'prompt' - how could this be initialised?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-16. Score: 1
  
  I believe you may want to use a [Sum Product Network](https://arxiv.org/pdf/1202.3732.pdf) for this task. SPNs are the state-of-the-art approach for face completion, and there are several more [recent papers](https://arxiv.org/pdf/1902.06155.pdf) on this topic since the original above.
  
  Importantly, the SPN paper also covers other approaches that work well for this task. If lower-resolution results are acceptable for your task, PCA with 100 or more components works surprisingly well. If your dataset is very large, a nearest-neighbor approach can work decently too.
  
  > **Comments:**
  > **username_0**: Thanks for your response. Please see my comment response to @ashenoy: could you please explain how to deal with some of the issues/concerns raised there about StyleGAN in terms of SPNs?  
  
  ---
  
  