I’m not sure about the specifics of your project. But from what you described, I feel like what you want is to un-project the depth map (using whatever camera parameters) to a 3D point cloud. If that is the case, I think you would want graph CNNs / pointnet / pointCNN.