Answers.Body: <p>It should be possible with a combination of object detection, monocular depth estimation (for example <a href="https://arxiv.org/abs/1806.01260" rel="nofollow noreferrer">Monodepth, Godard et al.</a>) and some math. Here is an example of how this could look like:</p>
  <ol>
  <li>Apply <strong>object detection/segmentation</strong> model to your image to find bounding boxes</li>
  <li>Determine the position of the center of the bounding box in the camera FOV, you could calculate a relative position here via: <span class="math-container">$\frac{\text{bbox}_x}{\text{img-width}} = p_x$</span></li>
  <li>Apply <strong>monocular depth estimation</strong> to generate a depth map from your camera image</li>
  <li>Take the bounding boxes to cut out regions in the depth maps and compute the average distance of the object to the camera lens.</li>
  </ol>
  <p>Now you have: relative x-position in FOV <span class="math-container">$p_x$</span> and distance <span class="math-container">$d$</span>, which gives you a vector in radial coordinates from your camera lens <span class="math-container">$\mathbf{\hat{p}} = (p_x, d)$</span> that directly translates to a position in the first image you showed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T08:02:36.710Z
Answers.Text: That's what I thought at first, this approach looks very straightforward and doable, but is very complex if we want to use different cameras.
  I failed to highlight in my question that I am looking for a little bit more exotic approach - is possible to estimate object location by surrounding objects in an image on a 2D map? For example:
  We can see a car near the corner of the building. Can we recognize that building on the map, locate it, and estimate that the car is near the building corner?
Answers.UserDisplayName: user3500960
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T09:30:53.280Z
Answers.Text: Ah, that will be more difficult. Here is something to think about first though: If your biggest problem is the changing cameras, I'd go for something like a CNN that estimates the FOV for each camera to make the approach in my answer applicable. This way would be simpler and would most likely also generalize better. But sounds like that's also not exactly what you are after. What do you think of the FOV estimation to account for changing cams? Or is it more that you'd like an "end-to-end AI solution" for this problem?
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T10:48:11.410Z
Answers.Text: I think FOV estimation is a smart solution and I will try to build something using this approach.
  However having "end-to-end" solution would be amazing. I am still not very advanced in this field, so building something from scratch is still too complex for me.
Answers.UserDisplayName: user3500960
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T14:30:56.423Z
Answers.Text: Okay, IMO an end-to-end solution would have to 'understand' the concept of a top-down view and also the concept of stationary vs mobile objects. Then, the camera image can be matched to all stationary objects that are also visible from top-down on the map as the reference points. However, this will not be easy to implement reliably and with the ability to generalize to unseen scenery.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T14:31:15.677Z
Answers.Text: I'd try first with some modular AI system which should be easier to do and presents more structure on how to approach this. On the other hand you basically have a mountain of available AI models at your disposal but with no real 'recipe' how to apply it to your case, it can be quite frustrating when you are new to the field. Maybe some related area that deals with somewhat similar problems is SLAM, i.e. simultaneous locating and mapping - it sounds like its very related to your project, maybe you find some inspiration there.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-20T15:28:18.903Z
Answers.LastActivityDate: 2024-02-20T15:33:29.400Z
Answers.LastEditDate: 2024-02-20T15:33:29.400Z
Answers.UserDisplayName: Chillston
Body: <p>I have been researching if there are any existing machine learning models that would help mark objects (for example: cars) on the map having only image, camera location, and camera orientation.</p>
  <p>For example:</p>
  <p><a href="https://i.sstatic.net/aTJZQ.jpg" rel="nofollow noreferrer">We know the camera's location, FOV</a></p>
  <p><a href="https://i.sstatic.net/R8xke.jpg" rel="nofollow noreferrer">From the camera's image we detect objects: 2 cars</a></p>
  <p><a href="https://i.sstatic.net/Weff9.jpg" rel="nofollow noreferrer">Our model having all the info provided before marks those two cars on the map</a></p>
  <p>Are there any existing models that could help solve this problem? With today's tools is it even possible to build such a model?</p>
  
Tags: <neural-networks><machine-learning><computer-vision>
Title: Marking object on a map from the image
UserDisplayName: user3500960
ThreadText: # Marking object on a map from the image
  
  (Asked by: username_0 on 2024-02-20)
  
  I have been researching if there are any existing machine learning models that would help mark objects (for example: cars) on the map having only image, camera location, and camera orientation.
  
  For example:
  
  [We know the camera's location, FOV](https://i.sstatic.net/aTJZQ.jpg)
  
  [From the camera's image we detect objects: 2 cars](https://i.sstatic.net/R8xke.jpg)
  
  [Our model having all the info provided before marks those two cars on the map](https://i.sstatic.net/Weff9.jpg)
  
  Are there any existing models that could help solve this problem? With today's tools is it even possible to build such a model?
  
  **Tags:** neural-networks, machine-learning, computer-vision
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-20. Score: 2
  
  It should be possible with a combination of object detection, monocular depth estimation (for example [Monodepth, Godard et al.](https://arxiv.org/abs/1806.01260)) and some math. Here is an example of how this could look like:
  
  1. Apply **object detection/segmentation** model to your image to find bounding boxes
  2. Determine the position of the center of the bounding box in the camera FOV, you could calculate a relative position here via: $\frac{\text{bbox}\_x}{\text{img-width}} = p\_x$
  3. Apply **monocular depth estimation** to generate a depth map from your camera image
  4. Take the bounding boxes to cut out regions in the depth maps and compute the average distance of the object to the camera lens.
  
  Now you have: relative x-position in FOV $p\_x$ and distance $d$, which gives you a vector in radial coordinates from your camera lens $\mathbf{\hat{p}} = (p\_x, d)$ that directly translates to a position in the first image you showed.
  
  > **Comments:**
  > **username_0**: That's what I thought at first, this approach looks very straightforward and doable, but is very complex if we want to use different cameras. I failed to highlight in my question that I am looking for a little bit more exotic approach - is possible to estimate object location by surrounding objects in an image on a 2D map? For example: We can see a car near the corner of the building. Can we recognize that building on the map, locate it, and estimate that the car is near the building corner?  
  > **username_1**: Ah, that will be more difficult. Here is something to think about first though: If your biggest problem is the changing cameras, I'd go for something like a CNN that estimates the FOV for each camera to make the approach in my answer applicable. This way would be simpler and would most likely also generalize better. But sounds like that's also not exactly what you are after. What do you think of the FOV estimation to account for changing cams? Or is it more that you'd like an "end-to-end AI solution" for this problem?  
  > **username_0**: I think FOV estimation is a smart solution and I will try to build something using this approach. However having "end-to-end" solution would be amazing. I am still not very advanced in this field, so building something from scratch is still too complex for me.  
  > **username_1**: Okay, IMO an end-to-end solution would have to 'understand' the concept of a top-down view and also the concept of stationary vs mobile objects. Then, the camera image can be matched to all stationary objects that are also visible from top-down on the map as the reference points. However, this will not be easy to implement reliably and with the ability to generalize to unseen scenery.  
  > **username_1**: I'd try first with some modular AI system which should be easier to do and presents more structure on how to approach this. On the other hand you basically have a mountain of available AI models at your disposal but with no real 'recipe' how to apply it to your case, it can be quite frustrating when you are new to the field. Maybe some related area that deals with somewhat similar problems is SLAM, i.e. simultaneous locating and mapping - it sounds like its very related to your project, maybe you find some inspiration there.  
  
  ---
  
  