I think that the fastest approach would be do process each projection with conv2d to extract the edges of the shape. these would create the vertices of the shape.
Then reconstruct the 3d shape by creating faces linking the vertices found in each projection.
although it may work for polyhedra. I wonder how it will do for spheroids.
And I have no idea on how to tackle this with nn.
maybe this step could be done easily with a simple algebraic algorithm.
I will probe the stackoverflow forums and math forums.