Answers.Body: <p>Probably closely related to the problem of interest would be
  something akin Neural Radiance Fields (NeRF for short) <a href="https://www.matthewtancik.com/nerf" rel="nofollow noreferrer">https://www.matthewtancik.com/nerf</a>.</p>
  <p>The model takes several images from different angles and view of the scene and learns a 3d representation of the scene, that can be used to sample novel views of this scene (not present in the training data).</p>
  <p><a href="https://i.sstatic.net/QrKv4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QrKv4.png" alt="enter image description here" /></a></p>
  <p>The knowledge of the scene is encoded in the weights of MLP. Namely, the renderer is a function that takes coordinates <span class="math-container">$(x, y, z)$</span> and viewing angle <span class="math-container">$(\theta, \phi)$</span> as an input, and outputs the color (RGB) and the density <span class="math-container">$\sigma$</span>.</p>
  <p><a href="https://i.sstatic.net/DXKmQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DXKmQ.png" alt="From" /></a></p>
  <p>For your problem, one can modify, the network, that is takes in addition to the coordinates and angles, other properties of the scene, such as lightning (some scalar), e.t.c and outputs again (RGB) vector and density <span class="math-container">$\sigma$</span>.</p>
  <p>In order for procedure to be successful, I would expect, that model needs data with different lighting, and provided there are enough data points, one could synthesize the view with given lightning.</p>
  <p>The presence of multiple objects can be probably taken into account, by passing a set of coordinates <span class="math-container">$(x_i, y_i, z_i)$</span> - but this seems to complicate things a lot, and the training procedure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-24T06:25:05.413Z
Answers.Text: This is very close to what I need indeed, with the different output, I think their model outputs some data like points in cloud, and only later they make a scene from this by use of traditional rendering methods. What I want to do is to make such a renderer to produce ready 2D images directly from input data. But I think I will try something like reversed FC network or reversed CNN network, that instead of making predictions from images it makes images from predictions, sort of thing.
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-23T14:17:58.470Z
Answers.LastActivityDate: 2021-07-23T14:17:58.470Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Is it possible to train a DL model that will generate a full resolution 2D image based on few numbers describing this image and what type of model or architecture would that be?
  <a href="https://i.sstatic.net/uPCXU.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uPCXU.jpg" alt="enter image description here" /></a></p>
  <p>What I want to achieve is that I deliver to the model some numbers for example describing positions of objects on the screen and number describing how lit the scene is and I get back a 2D image with objects in their correct positions and proper lighting, but for one set of input data values I will get always one same image (see image above). These input data also could be anything else than positions and lighting, these are only examples helping to visualize what I mean.</p>
  <p>This all, of course, assuming that I have a lot of annotated training data that consists of images and labels of the objects' positions and scene lighting values.</p>
  <p><strong>EDIT:</strong> The final model would be trained on real images taken from Full HD camera, not some simple shapes like presented here, that I did only to explain better my question.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-23T09:08:37.717Z
Comments.Text: I do not see the reason to refer to any kind of deep learning for this problem. You have objects and properties of the scene and just render the scene and output view from a certain angle.
Comments.UserDisplayName: spiridon_the_sun_rotator
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-23T11:01:20.003Z
Comments.Text: @spiridon_the_sun_rotator Well, the reason is that this is only simplification of my problem that I done to help visualise what I am asking for. Later, I want to train suggested model on data of real images taken with Full HD camera and make sort of a realistic scene renderer, generated by DL.
Comments.UserDisplayName: GKozinski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-23T11:05:33.353Z
Comments.Text: Also, I don't see why, for experimental/research reasons, I should restrain myself from using other methods of doing tasks that can be done with regular/simple methods. My question was about whether such DL models exist, not how to render scene.
Comments.UserDisplayName: GKozinski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-23T18:00:05.610Z
Comments.Text: Did you take a look at GANs?
Comments.UserDisplayName: Aray Karjauv
Tags: <deep-learning><convolutional-neural-networks><computer-vision><generative-model><image-generation>
Title: Is it possible to use deep learning to generate a 2D image from a few numerical values?
UserDisplayName: GKozinski
ThreadText: # Is it possible to use deep learning to generate a 2D image from a few numerical values?
  
  (Asked by: username_0 on 2021-07-23)
  
  Is it possible to train a DL model that will generate a full resolution 2D image based on few numbers describing this image and what type of model or architecture would that be?
  [![enter image description here](https://i.sstatic.net/uPCXU.jpg)](https://i.sstatic.net/uPCXU.jpg)
  
  What I want to achieve is that I deliver to the model some numbers for example describing positions of objects on the screen and number describing how lit the scene is and I get back a 2D image with objects in their correct positions and proper lighting, but for one set of input data values I will get always one same image (see image above). These input data also could be anything else than positions and lighting, these are only examples helping to visualize what I mean.
  
  This all, of course, assuming that I have a lot of annotated training data that consists of images and labels of the objects' positions and scene lighting values.
  
  **EDIT:** The final model would be trained on real images taken from Full HD camera, not some simple shapes like presented here, that I did only to explain better my question.
  
  **Tags:** deep-learning, convolutional-neural-networks, computer-vision, generative-model, image-generation
  > **Comments:**
  > **username_1**: I do not see the reason to refer to any kind of deep learning for this problem. You have objects and properties of the scene and just render the scene and output view from a certain angle.  
  > **username_0**: @spiridon_the_sun_rotator Well, the reason is that this is only simplification of my problem that I done to help visualise what I am asking for. Later, I want to train suggested model on data of real images taken with Full HD camera and make sort of a realistic scene renderer, generated by DL.  
  > **username_0**: Also, I don't see why, for experimental/research reasons, I should restrain myself from using other methods of doing tasks that can be done with regular/simple methods. My question was about whether such DL models exist, not how to render scene.  
  > **username_2**: Did you take a look at GANs?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-23. Score: 0 | Accepted answer
  
  Probably closely related to the problem of interest would be
  something akin Neural Radiance Fields (NeRF for short) <https://www.matthewtancik.com/nerf>.
  
  The model takes several images from different angles and view of the scene and learns a 3d representation of the scene, that can be used to sample novel views of this scene (not present in the training data).
  
  [![enter image description here](https://i.sstatic.net/QrKv4.png)](https://i.sstatic.net/QrKv4.png)
  
  The knowledge of the scene is encoded in the weights of MLP. Namely, the renderer is a function that takes coordinates $(x, y, z)$ and viewing angle $(\theta, \phi)$ as an input, and outputs the color (RGB) and the density $\sigma$.
  
  [![From](https://i.sstatic.net/DXKmQ.png)](https://i.sstatic.net/DXKmQ.png)
  
  For your problem, one can modify, the network, that is takes in addition to the coordinates and angles, other properties of the scene, such as lightning (some scalar), e.t.c and outputs again (RGB) vector and density $\sigma$.
  
  In order for procedure to be successful, I would expect, that model needs data with different lighting, and provided there are enough data points, one could synthesize the view with given lightning.
  
  The presence of multiple objects can be probably taken into account, by passing a set of coordinates $(x\_i, y\_i, z\_i)$ - but this seems to complicate things a lot, and the training procedure.
  
  > **Comments:**
  > **username_0**: This is very close to what I need indeed, with the different output, I think their model outputs some data like points in cloud, and only later they make a scene from this by use of traditional rendering methods. What I want to do is to make such a renderer to produce ready 2D images directly from input data. But I think I will try something like reversed FC network or reversed CNN network, that instead of making predictions from images it makes images from predictions, sort of thing.  
  
  ---
  
  