Answers.Body: <p>In the end I decided to just learn how Blender's scripting works and wrote a script. I know there would be simpler ways to calculate the transformations but I'm no maths expert and I wanted to spend my time debugging my logic errors rather my maths errors.</p>
  
  <p>Basically what this script does is take 3 input points (in the form of empties) relative to the photogrammetry mesh and 3 input points in the same position relative to the tracking data, then attempts to line them up with each other, bringing the photogrammetry mesh along with it. This actually works pretty well in my testing. It's obviously not the fastest workflow but it does work and that's the important part.</p>
  
  <pre class="lang-python prettyprint-override"><code>import bpy
  import bmesh
  from mathutils import Matrix
  
  #####################################
  # TO USE:
  #
  # 1) create empties at three easily-identified positions in tracking data, called ea1, ea2 and ea3
  # 2) create empties at the same three positions but in photogrammetry data, called eb1, eb2 and eb3
  #    NOTE: eb1, eb2 and eb3 must be positioned IN THE SAME ORDER as ea1, ea2 and ea3!
  # 3) name the photogrammetry mesh "photogrammetry"
  # 4) run the script and be amazed
  #
  #####################################
  
  # from https://blender.stackexchange.com/a/118783/41192
  def createLookRotation(forward, up):
      rot = Matrix.Identity(3)
      rot[0] = up                 # x
      rot[1] = forward.cross(up)  # y
      rot[2] = forward            # z
      rot = rot.transposed()
      return rot.to_quaternion()
  
  def doMeshThing(obj, mesh, empty1, empty2, empty3):
      bm = bmesh.new()
      bm.verts.new(empty1.location)
      bm.verts.new(empty2.location)
      bm.verts.new(empty3.location)
      bm.verts.ensure_lookup_table()
      bm.faces.new([bm.verts[0], bm.verts[1], bm.verts[2]])
      bm.normal_update()
      bm.to_mesh(mesh)
      bpy.ops.object.select_all(action='DESELECT')
      obj.select = True
      bpy.ops.object.origin_set(type='ORIGIN_GEOMETRY')
      bm.faces.ensure_lookup_table()
      normal = bm.faces[0].normal
      guide = empty2.location - empty1.location
      quatRot = createLookRotation(normal, guide)
      obj.rotation_mode = 'QUATERNION'
      obj.rotation_quaternion = quatRot.inverted()
      bpy.ops.object.transform_apply(location=False, rotation=True, scale=False)
      obj.rotation_quaternion = quatRot
      bm.free()
  
  ea1 = bpy.data.objects['ea1']
  ea2 = bpy.data.objects['ea2']
  ea3 = bpy.data.objects['ea3']
  eb1 = bpy.data.objects['eb1']
  eb2 = bpy.data.objects['eb2']
  eb3 = bpy.data.objects['eb3']
  
  mesha = bpy.data.meshes.new('mesha')
  meshb = bpy.data.meshes.new('meshb')
  
  obja = bpy.data.objects.new('obja', mesha)
  objb = bpy.data.objects.new('objb', meshb)
  
  scene = bpy.context.scene
  scene.objects.link(obja)
  scene.objects.link(objb)
  
  doMeshThing(obja, mesha, ea1, ea2, ea3)
  doMeshThing(objb, meshb, eb1, eb2, eb3)
  
  bpy.context.scene.update()
  
  # position and rotate objb (photogrammetry) to match position and rotation of obja (tracking)
  photogrammetryMesh = bpy.data.objects['photogrammetry']
  photogrammetryMesh.parent = objb
  photogrammetryMesh.matrix_parent_inverse = objb.matrix_world.inverted()
  
  objb.location = obja.location
  objb.rotation_quaternion = obja.rotation_quaternion
  
  # calculate average dimension difference here and scale
  scaleDiffX = obja.dimensions.x / objb.dimensions.x
  scaleDiffY = obja.dimensions.y / objb.dimensions.y
  scaleDiffZ = obja.dimensions.z / objb.dimensions.z
  #averageScaleDiff = (scaleDiffX + scaleDiffY + scaleDiffZ) / 3
  averageScaleDiff = (scaleDiffX + scaleDiffY) / 2 # seems to be in local space, so z is useless
  objb.scale.x = averageScaleDiff
  objb.scale.y = averageScaleDiff
  objb.scale.z = averageScaleDiff
  
  bpy.context.scene.update()
  
  parented_wm = photogrammetryMesh.matrix_world.copy()
  photogrammetryMesh.parent = None
  photogrammetryMesh.matrix_world = parented_wm
  
  bpy.ops.object.select_all(action='DESELECT')
  obja.select = True
  bpy.ops.object.delete()
  bpy.ops.object.select_all(action='DESELECT')
  objb.select = True
  bpy.ops.object.delete()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-13T12:25:14.177Z
Answers.LastActivityDate: 2020-01-13T12:25:14.177Z
Answers.UserDisplayName: Clonkex
Body: <p>I have some footage, which I've tracked in After Effects. I've exported that 3D tracked camera to C4D Lite. I then took photos of the filming location and used Meshroom (a photogrammetry tool) to generate a mesh from the photos.</p>
  
  <p>Now I want to line up the photogrammetry mesh in C4D with nulls that I've placed relative to the tracking points in After Effects. What's the best way to do this? Or is there a better workflow to get a photogrammetry mesh lined up with tracked footage?</p>
  
  <p><strong>More info:</strong> Specifically, after watching Corridor's real-life portals video I decided to try to create my own portal effect. My plan is to take a rough photogrammetry mesh of the filming area. I would then track the camera in After Effects, somehow get that tracking data into a 3D modelling program, load the photogrammetry mesh in, line everything up so the tracked camera moves through the mesh as it does in the footage then rotate the camera and its animation 180° around the point where the portal mask is tracked onto the footage in AE. This would create a perfect replica of the camera's original motion, but looking out at the world from a different angle.</p>
  
  <p>As such I can't just use the footage as the photogrammetry input, as I specifically need the photogrammetry mesh to include unseen areas of the world.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T08:19:10.190Z
Comments.Text: Ideally you would have used images of the footage in mushroom too then your solutions would converge and the locations of the camera positions would have been solved in one go. (photogramettery is sort of automated tracking)
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T09:01:03.177Z
Comments.Text: @joojaa I'm not really sure what you mean. I'd still just get a mesh with no particular way to line it up with the camera tracking from AE. As in, I would have no way to get the scale, rotation and position of the mesh correct relative to the tracked camera.
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T09:08:31.173Z
Comments.Text: No you get a mesh and postion of the cameras. See mushroom solves the camera position in the first pass.
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:10:47.123Z
Comments.Text: @joojaa Well sure it solves the camera positions, but how would that be useful? I'm just not seeing the practical connection. I can't export the camera positions, and even if I could, I couldn't easily turn those positions into an animated camera in a 3D modelling program or After Effects. Plus the results of photogrammetry from video would be far inferior to those from photos, since sharp images give drastically better results.
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:11:22.033Z
Comments.Text: Positions are animation
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:29:40.783Z
Comments.Text: @joojaa You seem very confident of this process, but I still don't see how it could be useful. The positions are _not_ animation on their own. Please explain how you would have 3D modelling software interpret a series of positions as animation unless they were already defined as such.
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:34:35.327Z
Comments.Text: Well i dont use C4D but i have done this in Maya, Blender, and 3DS max several times. Just loop over the objects and keyframe one other camera to their potions.
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:42:49.773Z
Comments.Text: @joojaa I have no particular interest in using C4D if I can use Blender, but I assumed this would be something other people do commonly with C4D Lite and AE so I only mentioned C4D. Perhaps you would like to write a more in-depth answer? Because "loop over the objects" doesn't sound like something you "just" do. Unless you are implying I should write a script to do this? But aside from that, that ultimately doesn't solve my problem. I'll update my question to be more specific.
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:47:17.843Z
Comments.Text: Well everything seems overly complex if you dont know anything about it
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:49:21.727Z
Comments.Text: Anyway you should be using video production stackexchange. Also meshroom can export camera as animation
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:54:42.707Z
Comments.Text: @joojaa It seems overly complex because it _should_ be overly complex. As in, there should be no reason to have to go to that much effort. It essentially boils down to needing a way to make a mesh in Blender (or C4D) line up based on 3 or 4 empties to another mesh with 3 or 4 empties, where the relative positions will never be quite precisely the same. I only now realise that's my actual problem. The difficulty comes in trying to search for such a thing...
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:56:16.680Z
Comments.Text: Its not allways possible to make 3 things work together without a bit of script glue. Its not much harder than writing for items selected take item copy attribute to other item
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:58:12.767Z
Comments.Text: @joojaa Any scripting is hard when you have to sift through a reference for every second line. I've never written a Blender script, except 10 years ago when I tried to make games with it. I could learn for sure but this is feeling like way more effort than it's worth.
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T10:58:58.710Z
Comments.Text: You can allways give up on your idea isnetad. But im sure somebody has wirtten this script
Comments.UserDisplayName: joojaa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T11:26:09.893Z
Comments.Text: @joojaa Oh I'm not giving up yet. https://blender.stackexchange.com/q/163459/41192 But it's definitely pretty tiresome when something seems like it should be so simple and easy but no one else even wanted to do the same thing so there's no help on it.
Comments.UserDisplayName: Clonkex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T12:04:06.493Z
Comments.Text: Your issue might be simpler than your question says. It does not particularity matter where the warp to object is from users point of view. just use meshroom to track scene 1 and scene 2. Export scene 1 as a camera animation. place yoru other stuff in that scene wherever you want it. make the portal in 3d render that. render scene 2.  do not track anything in AE. just comp the results so that 3d portal is alphamask. AE really does not do you any favors here.
Comments.UserDisplayName: joojaa
Tags: <after-effects><cinema4d>
Title: Line up photogrammetry mesh with tracking markers
UserDisplayName: Clonkex
ThreadText: # Line up photogrammetry mesh with tracking markers
  
  (Asked by: username_0 on 2020-01-11)
  
  I have some footage, which I've tracked in After Effects. I've exported that 3D tracked camera to C4D Lite. I then took photos of the filming location and used Meshroom (a photogrammetry tool) to generate a mesh from the photos.
  
  Now I want to line up the photogrammetry mesh in C4D with nulls that I've placed relative to the tracking points in After Effects. What's the best way to do this? Or is there a better workflow to get a photogrammetry mesh lined up with tracked footage?
  
  **More info:** Specifically, after watching Corridor's real-life portals video I decided to try to create my own portal effect. My plan is to take a rough photogrammetry mesh of the filming area. I would then track the camera in After Effects, somehow get that tracking data into a 3D modelling program, load the photogrammetry mesh in, line everything up so the tracked camera moves through the mesh as it does in the footage then rotate the camera and its animation 180° around the point where the portal mask is tracked onto the footage in AE. This would create a perfect replica of the camera's original motion, but looking out at the world from a different angle.
  
  As such I can't just use the footage as the photogrammetry input, as I specifically need the photogrammetry mesh to include unseen areas of the world.
  
  **Tags:** after-effects, cinema4d
  
  > **Comments:**
  > **username_1**: Ideally you would have used images of the footage in mushroom too then your solutions would converge and the locations of the camera positions would have been solved in one go. (photogramettery is sort of automated tracking)  
  > **username_0**: @username_1 I'm not really sure what you mean. I'd still just get a mesh with no particular way to line it up with the camera tracking from AE. As in, I would have no way to get the scale, rotation and position of the mesh correct relative to the tracked camera.  
  > **username_1**: No you get a mesh and postion of the cameras. See mushroom solves the camera position in the first pass.  
  > **username_0**: @username_1 Well sure it solves the camera positions, but how would that be useful? I'm just not seeing the practical connection. I can't export the camera positions, and even if I could, I couldn't easily turn those positions into an animated camera in a 3D modelling program or After Effects. Plus the results of photogrammetry from video would be far inferior to those from photos, since sharp images give drastically better results.  
  > **username_1**: Positions are animation  
  > **username_0**: @username_1 You seem very confident of this process, but I still don't see how it could be useful. The positions are _not_ animation on their own. Please explain how you would have 3D modelling software interpret a series of positions as animation unless they were already defined as such.  
  > **username_1**: Well i dont use C4D but i have done this in Maya, Blender, and 3DS max several times. Just loop over the objects and keyframe one other camera to their potions.  
  > **username_0**: @username_1 I have no particular interest in using C4D if I can use Blender, but I assumed this would be something other people do commonly with C4D Lite and AE so I only mentioned C4D. Perhaps you would like to write a more in-depth answer? Because "loop over the objects" doesn't sound like something you "just" do. Unless you are implying I should write a script to do this? But aside from that, that ultimately doesn't solve my problem. I'll update my question to be more specific.  
  > **username_1**: Well everything seems overly complex if you dont know anything about it  
  > **username_1**: Anyway you should be using video production stackexchange. Also meshroom can export camera as animation  
  > **username_0**: @username_1 It seems overly complex because it _should_ be overly complex. As in, there should be no reason to have to go to that much effort. It essentially boils down to needing a way to make a mesh in Blender (or C4D) line up based on 3 or 4 empties to another mesh with 3 or 4 empties, where the relative positions will never be quite precisely the same. I only now realise that's my actual problem. The difficulty comes in trying to search for such a thing...  
  > **username_1**: Its not allways possible to make 3 things work together without a bit of script glue. Its not much harder than writing for items selected take item copy attribute to other item  
  > **username_0**: @username_1 Any scripting is hard when you have to sift through a reference for every second line. I've never written a Blender script, except 10 years ago when I tried to make games with it. I could learn for sure but this is feeling like way more effort than it's worth.  
  > **username_1**: You can allways give up on your idea isnetad. But im sure somebody has wirtten this script  
  > **username_0**: @username_1 Oh I'm not giving up yet. https://blender.stackexchange.com/q/163459/41192 But it's definitely pretty tiresome when something seems like it should be so simple and easy but no one else even wanted to do the same thing so there's no help on it.  
  > **username_1**: Your issue might be simpler than your question says. It does not particularity matter where the warp to object is from users point of view. just use meshroom to track scene 1 and scene 2. Export scene 1 as a camera animation. place yoru other stuff in that scene wherever you want it. make the portal in 3d render that. render scene 2.  do not track anything in AE. just comp the results so that 3d portal is alphamask. AE really does not do you any favors here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-01-13. Score: 0 | Accepted answer
  
  In the end I decided to just learn how Blender's scripting works and wrote a script. I know there would be simpler ways to calculate the transformations but I'm no maths expert and I wanted to spend my time debugging my logic errors rather my maths errors.
  
  Basically what this script does is take 3 input points (in the form of empties) relative to the photogrammetry mesh and 3 input points in the same position relative to the tracking data, then attempts to line them up with each other, bringing the photogrammetry mesh along with it. This actually works pretty well in my testing. It's obviously not the fastest workflow but it does work and that's the important part.
  
  ```python
  import bpy
  import bmesh
  from mathutils import Matrix
  
  #####################################
  # TO USE:
  #
  # 1) create empties at three easily-identified positions in tracking data, called ea1, ea2 and ea3
  # 2) create empties at the same three positions but in photogrammetry data, called eb1, eb2 and eb3
  #    NOTE: eb1, eb2 and eb3 must be positioned IN THE SAME ORDER as ea1, ea2 and ea3!
  # 3) name the photogrammetry mesh "photogrammetry"
  # 4) run the script and be amazed
  #
  #####################################
  
  # from https://blender.stackexchange.com/a/118783/41192
  def createLookRotation(forward, up):
      rot = Matrix.Identity(3)
      rot[0] = up                 # x
      rot[1] = forward.cross(up)  # y
      rot[2] = forward            # z
      rot = rot.transposed()
      return rot.to_quaternion()
  
  def doMeshThing(obj, mesh, empty1, empty2, empty3):
      bm = bmesh.new()
      bm.verts.new(empty1.location)
      bm.verts.new(empty2.location)
      bm.verts.new(empty3.location)
      bm.verts.ensure_lookup_table()
      bm.faces.new([bm.verts[0], bm.verts[1], bm.verts[2]])
      bm.normal_update()
      bm.to_mesh(mesh)
      bpy.ops.object.select_all(action='DESELECT')
      obj.select = True
      bpy.ops.object.origin_set(type='ORIGIN_GEOMETRY')
      bm.faces.ensure_lookup_table()
      normal = bm.faces[0].normal
      guide = empty2.location - empty1.location
      quatRot = createLookRotation(normal, guide)
      obj.rotation_mode = 'QUATERNION'
      obj.rotation_quaternion = quatRot.inverted()
      bpy.ops.object.transform_apply(location=False, rotation=True, scale=False)
      obj.rotation_quaternion = quatRot
      bm.free()
  
  ea1 = bpy.data.objects['ea1']
  ea2 = bpy.data.objects['ea2']
  ea3 = bpy.data.objects['ea3']
  eb1 = bpy.data.objects['eb1']
  eb2 = bpy.data.objects['eb2']
  eb3 = bpy.data.objects['eb3']
  
  mesha = bpy.data.meshes.new('mesha')
  meshb = bpy.data.meshes.new('meshb')
  
  obja = bpy.data.objects.new('obja', mesha)
  objb = bpy.data.objects.new('objb', meshb)
  
  scene = bpy.context.scene
  scene.objects.link(obja)
  scene.objects.link(objb)
  
  doMeshThing(obja, mesha, ea1, ea2, ea3)
  doMeshThing(objb, meshb, eb1, eb2, eb3)
  
  bpy.context.scene.update()
  
  # position and rotate objb (photogrammetry) to match position and rotation of obja (tracking)
  photogrammetryMesh = bpy.data.objects['photogrammetry']
  photogrammetryMesh.parent = objb
  photogrammetryMesh.matrix_parent_inverse = objb.matrix_world.inverted()
  
  objb.location = obja.location
  objb.rotation_quaternion = obja.rotation_quaternion
  
  # calculate average dimension difference here and scale
  scaleDiffX = obja.dimensions.x / objb.dimensions.x
  scaleDiffY = obja.dimensions.y / objb.dimensions.y
  scaleDiffZ = obja.dimensions.z / objb.dimensions.z
  #averageScaleDiff = (scaleDiffX + scaleDiffY + scaleDiffZ) / 3
  averageScaleDiff = (scaleDiffX + scaleDiffY) / 2 # seems to be in local space, so z is useless
  objb.scale.x = averageScaleDiff
  objb.scale.y = averageScaleDiff
  objb.scale.z = averageScaleDiff
  
  bpy.context.scene.update()
  
  parented_wm = photogrammetryMesh.matrix_world.copy()
  photogrammetryMesh.parent = None
  photogrammetryMesh.matrix_world = parented_wm
  
  bpy.ops.object.select_all(action='DESELECT')
  obja.select = True
  bpy.ops.object.delete()
  bpy.ops.object.select_all(action='DESELECT')
  objb.select = True
  bpy.ops.object.delete()
  
  ```
  
  
  ---
  
  